[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:47 dist-test-slave-kkrh krb5kdc[9743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:47 dist-test-slave-kkrh krb5kdc[9743](info): set up 2 sockets
Sep 28 02:38:47 dist-test-slave-kkrh krb5kdc[9743](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 02:38:48.316744 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:38:48 dist-test-slave-kkrh krb5kdc[9743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:38:48.438917 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:34357
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:34357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:48.522722 9756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:48.522961 9756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:48.522998 9756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:48.524140 9756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:48.524206 9756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:48.524230 9756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:48.524257 9756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:48.524288 9756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:48.525705 9756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:34357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:34357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:48.525971 9756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:48.526193 9756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:48.530964 9762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:48.531016 9761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:48.531016 9764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:48.531178 9756 server_base.cc:1034] running on GCE node
I20260928 02:38:48.531464 9756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:48.531740 9756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:48.532898 9756 hybrid_clock.cc:648] HybridClock initialized: now 1790563128532893 us; error 53 us; skew 500 ppm
Sep 28 02:38:48 dist-test-slave-kkrh krb5kdc[9743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563128, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:48.535418 9756 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:38:48.535929 9756 webserver.cc:460] Webserver started at http://127.9.130.62:34835/ using document root <none> and password file <none>
I20260928 02:38:48.536159 9756 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:48.536226 9756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:48.536358 9756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:48.537372 9756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "12cc83ae957348c982631967972c585f"
format_stamp: "Formatted at 2026-09-28 02:38:48 on dist-test-slave-kkrh"
server_key: "49050f0de569870364557bf290acf81c"
server_key_iv: "c4251539bf068eaa221c77ca9401d3aa"
server_key_version: "encryptionkey@0"
I20260928 02:38:48.537744 9756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "12cc83ae957348c982631967972c585f"
format_stamp: "Formatted at 2026-09-28 02:38:48 on dist-test-slave-kkrh"
server_key: "49050f0de569870364557bf290acf81c"
server_key_iv: "c4251539bf068eaa221c77ca9401d3aa"
server_key_version: "encryptionkey@0"
I20260928 02:38:48.538983 9756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:48.539672 9771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:48.539855 9756 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:48.539959 9756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "12cc83ae957348c982631967972c585f"
format_stamp: "Formatted at 2026-09-28 02:38:48 on dist-test-slave-kkrh"
server_key: "49050f0de569870364557bf290acf81c"
server_key_iv: "c4251539bf068eaa221c77ca9401d3aa"
server_key_version: "encryptionkey@0"
I20260928 02:38:48.540057 9756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:48.561044 9756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:48.563726 9756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:48.563902 9756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:48.567582 9756 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:34357
I20260928 02:38:48.567636 9823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:34357 every 8 connection(s)
I20260928 02:38:48.568039 9756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:48.568920 9824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:48.571133 9824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f: Bootstrap starting.
I20260928 02:38:48.571688 9824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:48.572032 9824 log.cc:824] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:48.572691 9824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f: No bootstrap required, opened a new log
I20260928 02:38:48.574232 9824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cc83ae957348c982631967972c585f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34357 } }
I20260928 02:38:48.574373 9824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:48.574420 9824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12cc83ae957348c982631967972c585f, State: Initialized, Role: FOLLOWER
I20260928 02:38:48.574568 9824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cc83ae957348c982631967972c585f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34357 } }
I20260928 02:38:48.574626 9824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:48.574677 9824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:48.574728 9824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:48.575284 9824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cc83ae957348c982631967972c585f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34357 } }
I20260928 02:38:48.575397 9824 leader_election.cc:302] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12cc83ae957348c982631967972c585f; no voters:
I20260928 02:38:48.575577 9824 leader_election.cc:288] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:48.575610 9827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:48.575760 9827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [term 1 LEADER]: Becoming Leader. State: Replica: 12cc83ae957348c982631967972c585f, State: Running, Role: LEADER
I20260928 02:38:48.575858 9824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:48.575870 9827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cc83ae957348c982631967972c585f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34357 } }
I20260928 02:38:48.576422 9828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12cc83ae957348c982631967972c585f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cc83ae957348c982631967972c585f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34357 } } }
I20260928 02:38:48.576510 9828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:48.576480 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 9756
I20260928 02:38:48.576658 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:38:48.576701 9829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12cc83ae957348c982631967972c585f. Latest consensus state: current_term: 1 leader_uuid: "12cc83ae957348c982631967972c585f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cc83ae957348c982631967972c585f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34357 } } }
I20260928 02:38:48.576753 9829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:48.577009 9836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:48.577463 9736 external_mini_cluster.cc:1428] Setting key 632f2527cf43ad294e7f51d8ba86d236
I20260928 02:38:48.577517 9836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:48.579113 9836 catalog_manager.cc:1347] Generated new cluster ID: 9f56bfbc41ce4b5bbcffd40f3b351bd0
I20260928 02:38:48.579160 9836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:38:48 dist-test-slave-kkrh krb5kdc[9743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:48.595465 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:48.578374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.9.130.62:34357)
0928 02:38:48.578727 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:48.578736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:48.579884 (+ 1148us) server_negotiation.cc:407] Connection header received
0928 02:38:48.581309 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:48.581315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:48.581377 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:48.581514 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:48.583537 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:48.584921 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:48.586470 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:48.587047 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:48.587139 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:48.591394 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:48.591399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:48.591399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:48.591411 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:48.593450 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:48.593921 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:48.593924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:48.593924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:48.594038 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:48.594311 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:48.594312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:48.594312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:48.594642 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:48.594801 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:48.595083 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:38:48.595225 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":95,"threads_started":1}
I20260928 02:38:48.601999 9836 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:48.602735 9836 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:48.607304 9836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12cc83ae957348c982631967972c585f: Generated new TSK 0
I20260928 02:38:48.607489 9836 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:38:48.635401 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34357
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:48.715034 9850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:48.715225 9850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:48.715247 9850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:48.716428 9850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:48.716476 9850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:48.716502 9850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:38:48.717967 9850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34357
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:48.718221 9850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:48.718458 9850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:48.723440 9858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:48.723452 9855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:48.723469 9850 server_base.cc:1034] running on GCE node
W20260928 02:38:48.723452 9856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:48.723894 9850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:48.724094 9850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:48.725235 9850 hybrid_clock.cc:648] HybridClock initialized: now 1790563128725223 us; error 29 us; skew 500 ppm
Sep 28 02:38:48 dist-test-slave-kkrh krb5kdc[9743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563128, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:48.727664 9850 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:38:48.728114 9850 webserver.cc:460] Webserver started at http://127.9.130.1:34617/ using document root <none> and password file <none>
I20260928 02:38:48.728295 9850 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:48.728336 9850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:48.728420 9850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:48.729375 9850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "fced50453cb74d9cab51e04640a30bcd"
format_stamp: "Formatted at 2026-09-28 02:38:48 on dist-test-slave-kkrh"
server_key: "19f6d2e2b54858987e7967bdf05404a5"
server_key_iv: "a0d610d6365b2efcfa1be6632c51ab4c"
server_key_version: "encryptionkey@0"
I20260928 02:38:48.729734 9850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "fced50453cb74d9cab51e04640a30bcd"
format_stamp: "Formatted at 2026-09-28 02:38:48 on dist-test-slave-kkrh"
server_key: "19f6d2e2b54858987e7967bdf05404a5"
server_key_iv: "a0d610d6365b2efcfa1be6632c51ab4c"
server_key_version: "encryptionkey@0"
I20260928 02:38:48.730996 9850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:38:48.731626 9865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:48.731797 9850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:48.731922 9850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "fced50453cb74d9cab51e04640a30bcd"
format_stamp: "Formatted at 2026-09-28 02:38:48 on dist-test-slave-kkrh"
server_key: "19f6d2e2b54858987e7967bdf05404a5"
server_key_iv: "a0d610d6365b2efcfa1be6632c51ab4c"
server_key_version: "encryptionkey@0"
I20260928 02:38:48.732007 9850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:48.766589 9850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:48.769008 9850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:48.769169 9850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:48.769407 9850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:48.769719 9850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:48.769773 9850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:48.769816 9850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:48.769843 9850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:48.775589 9850 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:40227
I20260928 02:38:48.775717 9978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:40227 every 8 connection(s)
I20260928 02:38:48.775997 9850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:38:48.780275 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 9850
I20260928 02:38:48.780369 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:38:48.780530 9736 external_mini_cluster.cc:1428] Setting key 33dcf8c89f6272b254534d97da7e2e8f
Sep 28 02:38:48 dist-test-slave-kkrh krb5kdc[9743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563128, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:48.785550 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:48.776673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:36791 (local address 127.9.130.62:34357)
0928 02:38:48.776771 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:48.776774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:48.777001 (+ 227us) server_negotiation.cc:407] Connection header received
0928 02:38:48.777490 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:48.777491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:48.777501 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:48.777557 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:48.778418 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:48.778902 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:48.779905 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:48.780114 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:48.780167 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:48.782608 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:48.782609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:48.782610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:48.782617 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:48.784063 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:48.784559 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:48.784560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:48.784560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:48.784629 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:48.784922 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:48.784922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:48.784923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:48.785021 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:48.785083 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:48.785416 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 02:38:48.785485 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:38:48.785858 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:48.776663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34357 (local address 127.9.130.1:36791)
0928 02:38:48.776873 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:48.776908 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:48.777394 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:48.777610 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:48.777614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:48.778156 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:48.778341 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:48.778346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:48.779009 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:48.779010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:48.779776 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:48.779778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:48.780200 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:48.780201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:48.780351 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:48.780876 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:48.780884 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:48.782513 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:48.784139 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:48.784145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:48.784147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:48.784462 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:48.784702 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:48.784703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:48.784703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:48.784851 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:48.785099 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:48.785104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:48.785274 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 02:38:48.785400 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 02:38:48.785581 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260928 02:38:48.786479 9979 heartbeater.cc:347] Connected to a master server at 127.9.130.62:34357
I20260928 02:38:48.786574 9979 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:48.786765 9979 heartbeater.cc:511] Master 127.9.130.62:34357 requested a full tablet report, sending...
I20260928 02:38:48.787294 9788 ts_manager.cc:194] Registered new tserver with Master: fced50453cb74d9cab51e04640a30bcd (127.9.130.1:40227)
I20260928 02:38:48.787976 9788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:36791
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:38:48.806473 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34357
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:48.885959 9984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:48.886193 9984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:48.886231 9984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:48.887358 9984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:48.887431 9984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:48.887476 9984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:38:48.888908 9984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34357
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:48.889171 9984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:48.889393 9984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:48.894766 9989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:48.894766 9990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:48.894794 9992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:48.894804 9984 server_base.cc:1034] running on GCE node
I20260928 02:38:48.895226 9984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:48.895444 9984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:48.896632 9984 hybrid_clock.cc:648] HybridClock initialized: now 1790563128896615 us; error 27 us; skew 500 ppm
Sep 28 02:38:48 dist-test-slave-kkrh krb5kdc[9743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563128, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:48.899345 9984 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:38:48.899885 9984 webserver.cc:460] Webserver started at http://127.9.130.2:39329/ using document root <none> and password file <none>
I20260928 02:38:48.900105 9984 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:48.900175 9984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:48.900343 9984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:48.901420 9984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "77ac92c9d1f34b73b713bdd825a82a48"
format_stamp: "Formatted at 2026-09-28 02:38:48 on dist-test-slave-kkrh"
server_key: "748f9353ad95e87780a739350a445ac4"
server_key_iv: "21b42b6a370c1f7b9d22294d6a7a3255"
server_key_version: "encryptionkey@0"
I20260928 02:38:48.901849 9984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "77ac92c9d1f34b73b713bdd825a82a48"
format_stamp: "Formatted at 2026-09-28 02:38:48 on dist-test-slave-kkrh"
server_key: "748f9353ad95e87780a739350a445ac4"
server_key_iv: "21b42b6a370c1f7b9d22294d6a7a3255"
server_key_version: "encryptionkey@0"
I20260928 02:38:48.903240 9984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:48.904057 9999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:48.904217 9984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:48.904315 9984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "77ac92c9d1f34b73b713bdd825a82a48"
format_stamp: "Formatted at 2026-09-28 02:38:48 on dist-test-slave-kkrh"
server_key: "748f9353ad95e87780a739350a445ac4"
server_key_iv: "21b42b6a370c1f7b9d22294d6a7a3255"
server_key_version: "encryptionkey@0"
I20260928 02:38:48.904395 9984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:48.926151 9984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:48.928547 9984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:48.928714 9984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:48.928960 9984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:48.929280 9984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:48.929332 9984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:48.929366 9984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:48.929394 9984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:48.935680 9984 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:39637
I20260928 02:38:48.935739 10112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:39637 every 8 connection(s)
I20260928 02:38:48.936069 9984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:38:48.941365 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 9984
I20260928 02:38:48.941470 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:38:48.941632 9736 external_mini_cluster.cc:1428] Setting key 5ea5b97987bfc25daa8d131f206e70ee
Sep 28 02:38:48 dist-test-slave-kkrh krb5kdc[9743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563128, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:48.945785 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:48.936741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:43737 (local address 127.9.130.62:34357)
0928 02:38:48.936785 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:48.936792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:48.936979 (+ 187us) server_negotiation.cc:407] Connection header received
0928 02:38:48.937437 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:48.937438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:48.937447 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:48.937513 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:48.938375 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:48.938852 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:48.940011 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:48.940191 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:48.940244 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:48.942583 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:48.942584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:48.942584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:48.942592 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:48.944178 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:48.944661 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:48.944662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:48.944662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:48.944711 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:48.944998 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:48.944999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:48.944999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:48.945151 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:48.945225 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:48.945527 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 02:38:48.945615 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:38:48.945798 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:48.936706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34357 (local address 127.9.130.2:43737)
0928 02:38:48.936858 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:48.936897 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:48.937361 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:48.937566 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:48.937570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:48.938119 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:48.938301 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:48.938305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:48.939100 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:48.939101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:48.939936 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:48.939938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:48.940262 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:48.940263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:48.940354 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:48.940860 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:48.940905 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:48.942462 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:48.944255 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:48.944261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:48.944264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:48.944579 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:48.944775 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:48.944776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:48.944776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:48.944913 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:48.945241 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:48.945246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:48.945399 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 02:38:48.945483 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:38:48.945597 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260928 02:38:48.946530 10113 heartbeater.cc:347] Connected to a master server at 127.9.130.62:34357
I20260928 02:38:48.946636 10113 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:48.946826 10113 heartbeater.cc:511] Master 127.9.130.62:34357 requested a full tablet report, sending...
I20260928 02:38:48.947295 9788 ts_manager.cc:194] Registered new tserver with Master: 77ac92c9d1f34b73b713bdd825a82a48 (127.9.130.2:39637)
I20260928 02:38:48.947703 9788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:43737
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:38:48.967901 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34357
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:49.049154 10118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:49.049351 10118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:49.049374 10118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:49.050491 10118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:49.050530 10118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:49.050557 10118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:38:49.052009 10118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34357
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:49.052270 10118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:49.052502 10118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:49.057504 10126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:49.057510 10124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:49.057510 10123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:49.057797 10118 server_base.cc:1034] running on GCE node
I20260928 02:38:49.058014 10118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:49.058269 10118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:49.059465 10118 hybrid_clock.cc:648] HybridClock initialized: now 1790563129059445 us; error 33 us; skew 500 ppm
Sep 28 02:38:49 dist-test-slave-kkrh krb5kdc[9743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563129, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:49.062280 10118 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:38:49.062729 10118 webserver.cc:460] Webserver started at http://127.9.130.3:39395/ using document root <none> and password file <none>
I20260928 02:38:49.062989 10118 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:49.063064 10118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:49.063218 10118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:49.064399 10118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "0b683615816646919d479825c5e22849"
format_stamp: "Formatted at 2026-09-28 02:38:49 on dist-test-slave-kkrh"
server_key: "3a51cf64c796935b951cc984c31b257b"
server_key_iv: "402a5db7e7b46d97693d3774bbae7a18"
server_key_version: "encryptionkey@0"
I20260928 02:38:49.064775 10118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "0b683615816646919d479825c5e22849"
format_stamp: "Formatted at 2026-09-28 02:38:49 on dist-test-slave-kkrh"
server_key: "3a51cf64c796935b951cc984c31b257b"
server_key_iv: "402a5db7e7b46d97693d3774bbae7a18"
server_key_version: "encryptionkey@0"
I20260928 02:38:49.066179 10118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:38:49.066968 10133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:49.067236 10118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:49.067341 10118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "0b683615816646919d479825c5e22849"
format_stamp: "Formatted at 2026-09-28 02:38:49 on dist-test-slave-kkrh"
server_key: "3a51cf64c796935b951cc984c31b257b"
server_key_iv: "402a5db7e7b46d97693d3774bbae7a18"
server_key_version: "encryptionkey@0"
I20260928 02:38:49.067427 10118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:49.093645 10118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:49.096215 10118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:49.096385 10118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:49.096655 10118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:49.097021 10118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:49.097085 10118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:49.097132 10118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:49.097164 10118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:49.104002 10118 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:44149
I20260928 02:38:49.104045 10246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:44149 every 8 connection(s)
I20260928 02:38:49.104409 10118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:38:49 dist-test-slave-kkrh krb5kdc[9743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563129, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:49.112946 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 10118
I20260928 02:38:49.113037 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:38:49.113207 9736 external_mini_cluster.cc:1428] Setting key 107be54eedbcb971bf36e3aee9310f51
I20260928 02:38:49.113925 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:49.105156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:49731 (local address 127.9.130.62:34357)
0928 02:38:49.105225 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:49.105232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:49.105426 (+ 194us) server_negotiation.cc:407] Connection header received
0928 02:38:49.105893 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:49.105894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:49.105904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:49.106005 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:49.106836 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:49.107533 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:49.108659 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:49.108865 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:49.108927 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:49.110905 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:49.110906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:49.110906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:49.110913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:49.112476 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:49.112835 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:49.112836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:49.112836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:49.112885 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:49.113153 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:49.113154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:49.113154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:49.113261 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:49.113354 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:49.113740 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 02:38:49.113828 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:38:49.114138 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:49.105145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34357 (local address 127.9.130.3:49731)
0928 02:38:49.105302 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:49.105342 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:49.105811 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:49.106029 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:49.106032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:49.106572 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:49.106757 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:49.106762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:49.107615 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:49.107615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:49.108571 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:49.108574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:49.108941 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:49.108942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:49.109041 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:49.109486 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:49.109494 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:49.110827 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:49.112547 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:49.112551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:49.112552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:49.112746 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:49.112951 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:49.112952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:49.112952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:49.113091 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:49.113369 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:49.113375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:49.113574 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 02:38:49.113715 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:38:49.113883 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260928 02:38:49.114681 10247 heartbeater.cc:347] Connected to a master server at 127.9.130.62:34357
I20260928 02:38:49.114778 10247 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:49.114918 10247 heartbeater.cc:511] Master 127.9.130.62:34357 requested a full tablet report, sending...
I20260928 02:38:49.115271 9787 ts_manager.cc:194] Registered new tserver with Master: 0b683615816646919d479825c5e22849 (127.9.130.3:44149)
I20260928 02:38:49.115690 9787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:49731
I20260928 02:38:49.125669 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:38:49 dist-test-slave-kkrh krb5kdc[9743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563129, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:38:49 dist-test-slave-kkrh krb5kdc[9743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563129, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.130.1@KRBTEST.COM
I20260928 02:38:49.149513 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:49.141703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.9.130.1:40227)
0928 02:38:49.141856 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:49.141861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:49.141879 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:38:49.141906 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:49.141907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:49.141921 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:49.142079 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:49.142592 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:49.143480 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:49.144249 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:49.144582 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:49.144643 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:49.146923 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:49.146931 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:49.146931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:49.146938 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:49.148399 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:49.148737 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:49.148739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:49.148740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:49.148807 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:49.149017 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:49.149017 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:49.149018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:49.149156 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:49.149260 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:49.149324 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 02:38:49.149382 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
W20260928 02:38:49.149871 9893 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42646
Sep 28 02:38:49 dist-test-slave-kkrh krb5kdc[9743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:38:49 dist-test-slave-kkrh krb5kdc[9743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.1@KRBTEST.COM
I20260928 02:38:49.171592 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:49.164496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.9.130.1:40227)
0928 02:38:49.164566 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:49.164595 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:49.164668 (+ 73us) server_negotiation.cc:407] Connection header received
0928 02:38:49.164769 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:49.164770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:49.164783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:49.164850 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:49.165458 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:49.165961 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:49.166727 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:49.166933 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:49.167009 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:49.169320 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:49.169321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:49.169322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:49.169329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:49.170629 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:49.170940 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:49.170941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:49.170941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:49.170991 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:49.171161 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:49.171162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:49.171162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:49.171294 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:49.171395 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:49.171462 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 02:38:49.171523 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:38:49.172220 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 9850
I20260928 02:38:49.176942 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 9984
I20260928 02:38:49.181118 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 10118
I20260928 02:38:49.185405 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 9756
2026-09-28T02:38:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1909 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:49 dist-test-slave-kkrh krb5kdc[10261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:49 dist-test-slave-kkrh krb5kdc[10261](info): set up 2 sockets
Sep 28 02:38:49 dist-test-slave-kkrh krb5kdc[10261](info): commencing operation
krb5kdc: starting...
W20260928 02:38:50.228017 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:38:50 dist-test-slave-kkrh krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:38:50.346472 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:35613
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:35613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:50.425654 10274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:50.425897 10274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:50.425936 10274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:50.427070 10274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:50.427142 10274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:50.427166 10274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:50.427194 10274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:50.427225 10274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:50.428678 10274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:35613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:35613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:50.428941 10274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:50.429169 10274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:50.434067 10279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:50.434041 10282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:50.434077 10274 server_base.cc:1034] running on GCE node
W20260928 02:38:50.434058 10280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:50.434507 10274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:50.434860 10274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:50.436005 10274 hybrid_clock.cc:648] HybridClock initialized: now 1790563130435990 us; error 39 us; skew 500 ppm
Sep 28 02:38:50 dist-test-slave-kkrh krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563130, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:50.438516 10274 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:38:50.438966 10274 webserver.cc:460] Webserver started at http://127.9.130.62:36453/ using document root <none> and password file <none>
I20260928 02:38:50.439208 10274 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:50.439287 10274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:50.439443 10274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:50.440622 10274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "097fafb43af642b18249c8bef5037cb3"
format_stamp: "Formatted at 2026-09-28 02:38:50 on dist-test-slave-kkrh"
server_key: "908e0a401720a38494b9981a663f73bf"
server_key_iv: "56a74160a2841dbcd93f41d6cf7d44fc"
server_key_version: "encryptionkey@0"
I20260928 02:38:50.441017 10274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "097fafb43af642b18249c8bef5037cb3"
format_stamp: "Formatted at 2026-09-28 02:38:50 on dist-test-slave-kkrh"
server_key: "908e0a401720a38494b9981a663f73bf"
server_key_iv: "56a74160a2841dbcd93f41d6cf7d44fc"
server_key_version: "encryptionkey@0"
I20260928 02:38:50.442338 10274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:38:50.443053 10289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:50.443249 10274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:50.443351 10274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "097fafb43af642b18249c8bef5037cb3"
format_stamp: "Formatted at 2026-09-28 02:38:50 on dist-test-slave-kkrh"
server_key: "908e0a401720a38494b9981a663f73bf"
server_key_iv: "56a74160a2841dbcd93f41d6cf7d44fc"
server_key_version: "encryptionkey@0"
I20260928 02:38:50.443423 10274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:50.454905 10274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:50.456691 10274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:50.456857 10274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:50.460661 10274 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:35613
I20260928 02:38:50.460695 10341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:35613 every 8 connection(s)
I20260928 02:38:50.461042 10274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:50.461846 10342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:50.464097 10342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3: Bootstrap starting.
I20260928 02:38:50.464651 10342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:50.464949 10342 log.cc:824] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:50.465615 10342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3: No bootstrap required, opened a new log
I20260928 02:38:50.467154 10342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097fafb43af642b18249c8bef5037cb3" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35613 } }
I20260928 02:38:50.467290 10342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:50.467314 10342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 097fafb43af642b18249c8bef5037cb3, State: Initialized, Role: FOLLOWER
I20260928 02:38:50.467461 10342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097fafb43af642b18249c8bef5037cb3" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35613 } }
I20260928 02:38:50.467531 10342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:50.467552 10342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:50.467579 10342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:50.468196 10342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097fafb43af642b18249c8bef5037cb3" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35613 } }
I20260928 02:38:50.468304 10342 leader_election.cc:302] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 097fafb43af642b18249c8bef5037cb3; no voters:
I20260928 02:38:50.468477 10342 leader_election.cc:288] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:50.468521 10345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:50.468713 10345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [term 1 LEADER]: Becoming Leader. State: Replica: 097fafb43af642b18249c8bef5037cb3, State: Running, Role: LEADER
I20260928 02:38:50.468734 10342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:50.468796 10345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097fafb43af642b18249c8bef5037cb3" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35613 } }
I20260928 02:38:50.469257 10347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 097fafb43af642b18249c8bef5037cb3. Latest consensus state: current_term: 1 leader_uuid: "097fafb43af642b18249c8bef5037cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097fafb43af642b18249c8bef5037cb3" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35613 } } }
I20260928 02:38:50.469360 10347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:50.469255 10346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "097fafb43af642b18249c8bef5037cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097fafb43af642b18249c8bef5037cb3" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35613 } } }
I20260928 02:38:50.469497 10346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:50.469574 10350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:50.470119 10350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:50.471100 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 10274
I20260928 02:38:50.471194 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:38:50.471366 9736 external_mini_cluster.cc:1428] Setting key baa4206a3d0a89aebe93b2304c155995
I20260928 02:38:50.471666 10350 catalog_manager.cc:1347] Generated new cluster ID: e740da37b7284dd8b7bb6b3668d68fb3
I20260928 02:38:50.471727 10350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:50.475858 10350 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:50.476610 10350 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 02:38:50 dist-test-slave-kkrh krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:50.480930 10350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 097fafb43af642b18249c8bef5037cb3: Generated new TSK 0
I20260928 02:38:50.481103 10350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:50.482412 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.472013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.9.130.62:35613)
0928 02:38:50.472160 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:50.472169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:50.472207 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:38:50.472319 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:50.472326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:50.472380 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:50.472522 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:50.473184 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.474602 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.475905 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.476339 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.476470 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.478775 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:50.478779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:50.478780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:50.478792 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:50.480655 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:50.481089 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:50.481093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:50.481094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:50.481242 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:50.481494 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:50.481495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:50.481495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:50.481729 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:50.481831 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:50.482001 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 02:38:50.482119 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:38:50.515158 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:35613
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:50.613590 10368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:50.613816 10368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:50.613853 10368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:50.614995 10368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:50.615068 10368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:50.615115 10368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:38:50.616727 10368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:35613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:50.617013 10368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:50.617241 10368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:50.623127 10373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:50.623258 10376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:50.623369 10368 server_base.cc:1034] running on GCE node
W20260928 02:38:50.623169 10374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:50.623602 10368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:50.623875 10368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:50.625034 10368 hybrid_clock.cc:648] HybridClock initialized: now 1790563130625018 us; error 34 us; skew 500 ppm
Sep 28 02:38:50 dist-test-slave-kkrh krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563130, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:50.627693 10368 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:38:50.628255 10368 webserver.cc:460] Webserver started at http://127.9.130.1:44457/ using document root <none> and password file <none>
I20260928 02:38:50.628492 10368 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:50.628559 10368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:50.628708 10368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:50.629756 10368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "4d6e89af96a743c58848903c7157c6b7"
format_stamp: "Formatted at 2026-09-28 02:38:50 on dist-test-slave-kkrh"
server_key: "c640253b5651c3426a38432202ad02ef"
server_key_iv: "d6ae6e52fec5135730c18126531049bd"
server_key_version: "encryptionkey@0"
I20260928 02:38:50.630158 10368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "4d6e89af96a743c58848903c7157c6b7"
format_stamp: "Formatted at 2026-09-28 02:38:50 on dist-test-slave-kkrh"
server_key: "c640253b5651c3426a38432202ad02ef"
server_key_iv: "d6ae6e52fec5135730c18126531049bd"
server_key_version: "encryptionkey@0"
I20260928 02:38:50.631496 10368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:38:50.632331 10383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:50.632483 10368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:50.632565 10368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "4d6e89af96a743c58848903c7157c6b7"
format_stamp: "Formatted at 2026-09-28 02:38:50 on dist-test-slave-kkrh"
server_key: "c640253b5651c3426a38432202ad02ef"
server_key_iv: "d6ae6e52fec5135730c18126531049bd"
server_key_version: "encryptionkey@0"
I20260928 02:38:50.632629 10368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:50.649518 10368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:50.651978 10368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:50.652091 10368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:50.652377 10368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:50.652701 10368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:50.652734 10368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:50.652757 10368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:50.652801 10368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:50.658602 10368 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:38339
I20260928 02:38:50.658653 10496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:38339 every 8 connection(s)
I20260928 02:38:50.658947 10368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:38:50.660617 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 10368
I20260928 02:38:50.660697 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:38:50.660863 9736 external_mini_cluster.cc:1428] Setting key ec6a0f117c7be96840126908288728c5
Sep 28 02:38:50 dist-test-slave-kkrh krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563130, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
I20260928 02:38:50.673435 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.659619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:47553 (local address 127.9.130.62:35613)
0928 02:38:50.659706 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:50.659708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:50.659947 (+ 239us) server_negotiation.cc:407] Connection header received
0928 02:38:50.661007 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:50.661009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:50.661022 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:50.661069 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:50.661796 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.662290 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.663869 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.664079 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.664139 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.666499 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:50.666505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:50.666506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:50.666513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:50.671761 (+ 5248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:50.672303 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:50.672305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:50.672305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:50.672371 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:50.672635 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:50.672636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:50.672637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:50.672792 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:50.672870 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:50.673298 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 02:38:50.673369 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:38:50.673722 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.659582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:35613 (local address 127.9.130.1:47553)
0928 02:38:50.659771 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:50.659849 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:50.660882 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:50.661154 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:50.661157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:50.661502 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:50.661700 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:50.661705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.662383 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.662384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:50.663741 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:50.663744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.664164 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.664165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:50.664312 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.664797 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:50.664807 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:50.666390 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:50.671912 (+ 5522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:50.671920 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:50.671921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:50.672216 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:50.672446 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:50.672447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:50.672447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:50.672568 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:50.672890 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:50.672898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:50.673138 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 02:38:50.673273 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 02:38:50.673464 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260928 02:38:50.674348 10497 heartbeater.cc:347] Connected to a master server at 127.9.130.62:35613
I20260928 02:38:50.674469 10497 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:50.674644 10497 heartbeater.cc:511] Master 127.9.130.62:35613 requested a full tablet report, sending...
I20260928 02:38:50.675209 10306 ts_manager.cc:194] Registered new tserver with Master: 4d6e89af96a743c58848903c7157c6b7 (127.9.130.1:38339)
I20260928 02:38:50.675894 10306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:47553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:38:50.687697 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:35613
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:50.767144 10502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:50.767391 10502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:50.767427 10502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:50.768684 10502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:50.768743 10502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:50.768795 10502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:38:50.770453 10502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:35613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:50.770709 10502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:50.770926 10502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:50.775976 10507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:50.775983 10508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:50.776058 10502 server_base.cc:1034] running on GCE node
W20260928 02:38:50.775983 10510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:50.776363 10502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:50.776564 10502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:50.777698 10502 hybrid_clock.cc:648] HybridClock initialized: now 1790563130777690 us; error 31 us; skew 500 ppm
Sep 28 02:38:50 dist-test-slave-kkrh krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563130, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:50.780206 10502 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:38:50.780562 10502 webserver.cc:460] Webserver started at http://127.9.130.2:39665/ using document root <none> and password file <none>
I20260928 02:38:50.780793 10502 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:50.780860 10502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:50.781003 10502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:50.782104 10502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "40861cb7107141f5be9fa597880ba515"
format_stamp: "Formatted at 2026-09-28 02:38:50 on dist-test-slave-kkrh"
server_key: "f3367d00e1aa5a5f01aa4c3346539925"
server_key_iv: "cc549cacf36d7445e9ecf1349bafcb8a"
server_key_version: "encryptionkey@0"
I20260928 02:38:50.782478 10502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "40861cb7107141f5be9fa597880ba515"
format_stamp: "Formatted at 2026-09-28 02:38:50 on dist-test-slave-kkrh"
server_key: "f3367d00e1aa5a5f01aa4c3346539925"
server_key_iv: "cc549cacf36d7445e9ecf1349bafcb8a"
server_key_version: "encryptionkey@0"
I20260928 02:38:50.783782 10502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:50.784584 10517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:50.784751 10502 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:50.784845 10502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "40861cb7107141f5be9fa597880ba515"
format_stamp: "Formatted at 2026-09-28 02:38:50 on dist-test-slave-kkrh"
server_key: "f3367d00e1aa5a5f01aa4c3346539925"
server_key_iv: "cc549cacf36d7445e9ecf1349bafcb8a"
server_key_version: "encryptionkey@0"
I20260928 02:38:50.784927 10502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:50.802603 10502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:50.804898 10502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:50.805078 10502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:50.805315 10502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:50.805608 10502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:50.805663 10502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:50.805702 10502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:50.805729 10502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:50.811342 10502 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:39037
I20260928 02:38:50.811375 10630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:39037 every 8 connection(s)
I20260928 02:38:50.811693 10502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:38:50.812242 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 10502
I20260928 02:38:50.812315 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:38:50.812491 9736 external_mini_cluster.cc:1428] Setting key d91c572acb8070752b8066196c79b30f
Sep 28 02:38:50 dist-test-slave-kkrh krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563130, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
I20260928 02:38:50.822259 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.812466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:47157 (local address 127.9.130.62:35613)
0928 02:38:50.812514 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:50.812516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:50.812813 (+ 297us) server_negotiation.cc:407] Connection header received
0928 02:38:50.814048 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:50.814049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:50.814058 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:50.814108 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:50.814922 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.815773 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.816911 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.817231 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.817305 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.819397 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:50.819398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:50.819399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:50.819407 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:50.820967 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:50.821319 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:50.821320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:50.821321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:50.821370 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:50.821711 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:50.821712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:50.821712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:50.821822 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:50.821875 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:50.822135 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 02:38:50.822199 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:38:50.822528 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.812462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:35613 (local address 127.9.130.2:47157)
0928 02:38:50.812650 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:50.812698 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:50.813941 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:50.814179 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:50.814184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:50.814557 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:50.814826 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:50.814831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.815891 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.815892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:50.816805 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:50.816818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.817312 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.817312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:50.817409 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.817911 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:50.817919 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:50.819304 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:50.821047 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:50.821052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:50.821054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:50.821252 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:50.821461 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:50.821462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:50.821462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:50.821586 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:50.821889 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:50.821895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:50.822016 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 02:38:50.822111 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:38:50.822249 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
I20260928 02:38:50.823123 10631 heartbeater.cc:347] Connected to a master server at 127.9.130.62:35613
I20260928 02:38:50.823215 10631 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:50.823403 10631 heartbeater.cc:511] Master 127.9.130.62:35613 requested a full tablet report, sending...
I20260928 02:38:50.823858 10306 ts_manager.cc:194] Registered new tserver with Master: 40861cb7107141f5be9fa597880ba515 (127.9.130.2:39037)
I20260928 02:38:50.824229 10306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:47157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:38:50.838011 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:35613
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:50.917337 10636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:50.917533 10636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:50.917555 10636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:50.918708 10636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:50.918748 10636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:50.918777 10636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:38:50.920257 10636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:35613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:50.920470 10636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:50.920702 10636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:50.925678 10641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:50.925793 10636 server_base.cc:1034] running on GCE node
W20260928 02:38:50.925843 10644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:50.925700 10642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:50.926100 10636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:50.926333 10636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:50.927479 10636 hybrid_clock.cc:648] HybridClock initialized: now 1790563130927459 us; error 39 us; skew 500 ppm
Sep 28 02:38:50 dist-test-slave-kkrh krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563130, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:50.930111 10636 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:38:50.930512 10636 webserver.cc:460] Webserver started at http://127.9.130.3:34783/ using document root <none> and password file <none>
I20260928 02:38:50.930737 10636 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:50.930804 10636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:50.930949 10636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:50.932046 10636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "288a4eedbe5b4420b42fafe14d47e0e6"
format_stamp: "Formatted at 2026-09-28 02:38:50 on dist-test-slave-kkrh"
server_key: "e215eaa1fe81e6f0cf85b18a80dbc8bf"
server_key_iv: "682ce55d22e8783c634a46de319cb7e2"
server_key_version: "encryptionkey@0"
I20260928 02:38:50.932408 10636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "288a4eedbe5b4420b42fafe14d47e0e6"
format_stamp: "Formatted at 2026-09-28 02:38:50 on dist-test-slave-kkrh"
server_key: "e215eaa1fe81e6f0cf85b18a80dbc8bf"
server_key_iv: "682ce55d22e8783c634a46de319cb7e2"
server_key_version: "encryptionkey@0"
I20260928 02:38:50.933660 10636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:38:50.934410 10651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:50.934590 10636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:50.934697 10636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "288a4eedbe5b4420b42fafe14d47e0e6"
format_stamp: "Formatted at 2026-09-28 02:38:50 on dist-test-slave-kkrh"
server_key: "e215eaa1fe81e6f0cf85b18a80dbc8bf"
server_key_iv: "682ce55d22e8783c634a46de319cb7e2"
server_key_version: "encryptionkey@0"
I20260928 02:38:50.934773 10636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:50.946389 10636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:50.948482 10636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:50.948644 10636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:50.948895 10636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:50.949193 10636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:50.949230 10636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:50.949280 10636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:50.949307 10636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:50.955116 10636 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:40707
I20260928 02:38:50.955160 10764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:40707 every 8 connection(s)
I20260928 02:38:50.955451 10636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:38:50 dist-test-slave-kkrh krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563130, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:50.962669 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 10636
I20260928 02:38:50.962785 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:38:50.962962 9736 external_mini_cluster.cc:1428] Setting key c83fc08bd4abccdae5af9ba0aaf1e295
I20260928 02:38:50.964989 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.956144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:55771 (local address 127.9.130.62:35613)
0928 02:38:50.956236 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:50.956238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:50.956408 (+ 170us) server_negotiation.cc:407] Connection header received
0928 02:38:50.957312 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:50.957313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:50.957325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:50.957384 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:50.957949 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.958482 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.959612 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.959901 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.959985 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.962042 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:50.962044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:50.962044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:50.962052 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:50.963738 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:50.964185 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:50.964186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:50.964186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:50.964238 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:50.964455 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:50.964456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:50.964456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:50.964575 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:50.964643 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:50.964865 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 02:38:50.964927 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:38:50.965184 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.956093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:35613 (local address 127.9.130.3:55771)
0928 02:38:50.956287 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:50.956321 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:50.957205 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:50.957438 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:50.957442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:50.957677 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:50.957864 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:50.957868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.958581 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.958583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:50.959501 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:50.959504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.959995 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.959997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:50.960102 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.960565 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:50.960574 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:50.961937 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:50.963841 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:50.963847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:50.963850 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:50.964119 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:50.964282 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:50.964283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:50.964283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:50.964369 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:50.964635 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:50.964637 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:50.964754 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 02:38:50.964831 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:38:50.964948 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":115,"threads_started":1}
I20260928 02:38:50.965620 10765 heartbeater.cc:347] Connected to a master server at 127.9.130.62:35613
I20260928 02:38:50.965734 10765 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:50.965917 10765 heartbeater.cc:511] Master 127.9.130.62:35613 requested a full tablet report, sending...
I20260928 02:38:50.966252 10306 ts_manager.cc:194] Registered new tserver with Master: 288a4eedbe5b4420b42fafe14d47e0e6 (127.9.130.3:40707)
I20260928 02:38:50.966621 10306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:55771
I20260928 02:38:50.974808 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:38:50.982117 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.976342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.9.130.62:35613)
0928 02:38:50.976381 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:50.976383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:50.976490 (+ 107us) server_negotiation.cc:407] Connection header received
0928 02:38:50.976551 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:50.976551 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:50.976560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:50.976598 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:50.977127 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.977595 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.978281 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.978463 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.978512 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.979680 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:50.979681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:50.979681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:50.979686 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:50.981160 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:50.981517 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:50.981518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:50.981518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:50.981571 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:50.981719 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:50.981719 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:50.981719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:50.981842 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:50.981911 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:50.981997 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 02:38:50.982056 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:38:50.984427 10306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:38:50.985864 10306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:50.992657 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.988177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.9.130.3:40707)
0928 02:38:50.988380 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:50.988387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:50.988402 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:38:50.988478 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:50.988480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:50.988496 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:50.988828 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:50.989074 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.990487 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.991702 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.992223 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.992317 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.992376 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:50.992524 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 02:38:50.992564 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
I20260928 02:38:50.992645 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.988328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:38339 (local address 127.0.0.1:39396)
0928 02:38:50.988467 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:50.988476 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:50.988518 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:50.989063 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:50.989064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:50.989067 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:50.989125 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:50.989128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.990216 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.990217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:50.991371 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:50.991373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.992401 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.992402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:50.992479 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.992483 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:50.992517 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 02:38:50.992580 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":21,"thread_start_us":86,"threads_started":1}
I20260928 02:38:50.992800 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.988714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.9.130.1:38339)
0928 02:38:50.988900 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:50.988904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:50.988913 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:50.988949 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:50.988951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:50.988964 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:50.989199 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:50.989377 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.990036 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.991465 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.992297 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.992563 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.992627 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:50.992653 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 02:38:50.992694 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
I20260928 02:38:50.993148 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.988062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:40707 (local address 127.0.0.1:60394)
0928 02:38:50.988249 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:50.988293 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:50.988363 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:50.988926 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:50.988928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:50.988930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:50.988994 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:50.988996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.990605 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.990606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:50.991612 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:50.991613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.992319 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.992320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:50.992395 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.992408 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:38:50.992508 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:38:50.992840 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":121,"threads_started":1}
I20260928 02:38:50.993579 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.988504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:39037 (local address 127.0.0.1:41918)
0928 02:38:50.988649 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:50.988659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:50.988708 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:50.989074 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:50.989075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:50.989076 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:50.989125 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:50.989127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.990623 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.990624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:50.991796 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:50.991799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.993352 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.993353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:50.993418 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.993421 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:50.993475 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:38:50.993516 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260928 02:38:50.993527 10431 tablet_service.cc:1467] Processing CreateTablet for tablet 075a69c6ef664e17abbe483bd2182e9d (DEFAULT_TABLE table=test-table [id=f78c13f1667f4e9ea36f9a18c734de70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:50.993826 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:50.988647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.9.130.2:39037)
0928 02:38:50.988836 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:50.988841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:50.988851 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:38:50.988893 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:50.988895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:50.988911 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:50.988995 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:50.989292 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.990485 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.991929 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:50.993129 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:50.993251 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:50.993316 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:50.993706 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 02:38:50.993753 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260928 02:38:50.993872 10431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075a69c6ef664e17abbe483bd2182e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:50.993839 10699 tablet_service.cc:1467] Processing CreateTablet for tablet 075a69c6ef664e17abbe483bd2182e9d (DEFAULT_TABLE table=test-table [id=f78c13f1667f4e9ea36f9a18c734de70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:50.994107 10699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075a69c6ef664e17abbe483bd2182e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:50.994374 10565 tablet_service.cc:1467] Processing CreateTablet for tablet 075a69c6ef664e17abbe483bd2182e9d (DEFAULT_TABLE table=test-table [id=f78c13f1667f4e9ea36f9a18c734de70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:50.994614 10565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075a69c6ef664e17abbe483bd2182e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:50.997223 10782 tablet_bootstrap.cc:492] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515: Bootstrap starting.
I20260928 02:38:50.997893 10782 tablet_bootstrap.cc:654] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:50.997877 10784 tablet_bootstrap.cc:492] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7: Bootstrap starting.
I20260928 02:38:50.998399 10784 tablet_bootstrap.cc:654] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:50.998409 10782 log.cc:824] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:50.998915 10784 log.cc:824] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:50.999223 10783 tablet_bootstrap.cc:492] T 075a69c6ef664e17abbe483bd2182e9d P 288a4eedbe5b4420b42fafe14d47e0e6: Bootstrap starting.
I20260928 02:38:50.999305 10782 tablet_bootstrap.cc:492] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515: No bootstrap required, opened a new log
I20260928 02:38:50.999385 10782 ts_tablet_manager.cc:1397] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:50.999748 10783 tablet_bootstrap.cc:654] T 075a69c6ef664e17abbe483bd2182e9d P 288a4eedbe5b4420b42fafe14d47e0e6: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:51.000224 10783 log.cc:824] T 075a69c6ef664e17abbe483bd2182e9d P 288a4eedbe5b4420b42fafe14d47e0e6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:51.000687 10784 tablet_bootstrap.cc:492] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7: No bootstrap required, opened a new log
I20260928 02:38:51.000761 10784 ts_tablet_manager.cc:1397] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:38:51.001076 10783 tablet_bootstrap.cc:492] T 075a69c6ef664e17abbe483bd2182e9d P 288a4eedbe5b4420b42fafe14d47e0e6: No bootstrap required, opened a new log
I20260928 02:38:51.001150 10783 ts_tablet_manager.cc:1397] T 075a69c6ef664e17abbe483bd2182e9d P 288a4eedbe5b4420b42fafe14d47e0e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:51.001375 10782 raft_consensus.cc:348] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288a4eedbe5b4420b42fafe14d47e0e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 40707 } } peers { permanent_uuid: "4d6e89af96a743c58848903c7157c6b7" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38339 } } peers { permanent_uuid: "40861cb7107141f5be9fa597880ba515" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39037 } }
I20260928 02:38:51.001545 10782 raft_consensus.cc:374] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:51.001593 10782 raft_consensus.cc:729] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40861cb7107141f5be9fa597880ba515, State: Initialized, Role: FOLLOWER
I20260928 02:38:51.001713 10782 consensus_queue.cc:260] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288a4eedbe5b4420b42fafe14d47e0e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 40707 } } peers { permanent_uuid: "4d6e89af96a743c58848903c7157c6b7" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38339 } } peers { permanent_uuid: "40861cb7107141f5be9fa597880ba515" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39037 } }
I20260928 02:38:51.002007 10782 ts_tablet_manager.cc:1428] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:38:51.002161 10631 heartbeater.cc:503] Master 127.9.130.62:35613 was elected leader, sending a full tablet report...
I20260928 02:38:51.003095 10784 raft_consensus.cc:348] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288a4eedbe5b4420b42fafe14d47e0e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 40707 } } peers { permanent_uuid: "4d6e89af96a743c58848903c7157c6b7" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38339 } } peers { permanent_uuid: "40861cb7107141f5be9fa597880ba515" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39037 } }
I20260928 02:38:51.003264 10784 raft_consensus.cc:374] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:51.003310 10784 raft_consensus.cc:729] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d6e89af96a743c58848903c7157c6b7, State: Initialized, Role: FOLLOWER
I20260928 02:38:51.003422 10784 consensus_queue.cc:260] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288a4eedbe5b4420b42fafe14d47e0e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 40707 } } peers { permanent_uuid: "4d6e89af96a743c58848903c7157c6b7" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38339 } } peers { permanent_uuid: "40861cb7107141f5be9fa597880ba515" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39037 } }
I20260928 02:38:51.003402 10783 raft_consensus.cc:348] T 075a69c6ef664e17abbe483bd2182e9d P 288a4eedbe5b4420b42fafe14d47e0e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288a4eedbe5b4420b42fafe14d47e0e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 40707 } } peers { permanent_uuid: "4d6e89af96a743c58848903c7157c6b7" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38339 } } peers { permanent_uuid: "40861cb7107141f5be9fa597880ba515" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39037 } }
I20260928 02:38:51.003561 10783 raft_consensus.cc:374] T 075a69c6ef664e17abbe483bd2182e9d P 288a4eedbe5b4420b42fafe14d47e0e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:51.003603 10783 raft_consensus.cc:729] T 075a69c6ef664e17abbe483bd2182e9d P 288a4eedbe5b4420b42fafe14d47e0e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288a4eedbe5b4420b42fafe14d47e0e6, State: Initialized, Role: FOLLOWER
I20260928 02:38:51.003687 10784 ts_tablet_manager.cc:1428] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:38:51.003710 10783 consensus_queue.cc:260] T 075a69c6ef664e17abbe483bd2182e9d P 288a4eedbe5b4420b42fafe14d47e0e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288a4eedbe5b4420b42fafe14d47e0e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 40707 } } peers { permanent_uuid: "4d6e89af96a743c58848903c7157c6b7" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38339 } } peers { permanent_uuid: "40861cb7107141f5be9fa597880ba515" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39037 } }
I20260928 02:38:51.003847 10497 heartbeater.cc:503] Master 127.9.130.62:35613 was elected leader, sending a full tablet report...
I20260928 02:38:51.003968 10783 ts_tablet_manager.cc:1428] T 075a69c6ef664e17abbe483bd2182e9d P 288a4eedbe5b4420b42fafe14d47e0e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 02:38:51.004027 10765 heartbeater.cc:503] Master 127.9.130.62:35613 was elected leader, sending a full tablet report...
I20260928 02:38:51.023772 10789 raft_consensus.cc:482] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:51.023942 10789 raft_consensus.cc:504] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288a4eedbe5b4420b42fafe14d47e0e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 40707 } } peers { permanent_uuid: "4d6e89af96a743c58848903c7157c6b7" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38339 } } peers { permanent_uuid: "40861cb7107141f5be9fa597880ba515" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39037 } }
I20260928 02:38:51.024339 10789 leader_election.cc:288] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 288a4eedbe5b4420b42fafe14d47e0e6 (127.9.130.3:40707), 40861cb7107141f5be9fa597880ba515 (127.9.130.2:39037)
I20260928 02:38:51.027117 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:51.024507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:40707 (local address 127.9.130.1:44207)
0928 02:38:51.024554 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:51.024566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:51.024609 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:51.024793 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:51.024794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:51.024796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:51.024870 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:51.024872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.025561 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.025562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:51.026328 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:51.026330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.026892 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.026893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:51.026980 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:51.026983 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:51.027035 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:38:51.027065 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 02:38:51.027285 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:51.024596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:44207 (local address 127.9.130.3:40707)
0928 02:38:51.024651 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:51.024653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:51.024662 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:51.024688 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:51.024689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:51.024697 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:51.024786 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:51.024950 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.025484 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.026423 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.026819 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.026871 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:51.026879 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:51.027155 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 02:38:51.027197 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:38:51.027380 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:51.024757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:49095 (local address 127.9.130.2:39037)
0928 02:38:51.024798 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:51.024800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:51.024863 (+ 63us) server_negotiation.cc:407] Connection header received
0928 02:38:51.024910 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:51.024910 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:51.024923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:51.024973 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:51.025104 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.025573 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.026496 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.026883 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.026953 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:51.026964 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:51.027268 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 02:38:51.027310 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:38:51.027565 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:51.024611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:39037 (local address 127.9.130.1:49095)
0928 02:38:51.024791 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:51.024802 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:51.024858 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:51.024976 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:51.024977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:51.024978 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:51.025042 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:51.025043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.025650 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.025651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:51.026407 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:51.026409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.026977 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.026978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:51.027169 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:51.027173 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:51.027459 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 02:38:51.027506 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":110,"threads_started":1}
I20260928 02:38:51.027556 10719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075a69c6ef664e17abbe483bd2182e9d" candidate_uuid: "4d6e89af96a743c58848903c7157c6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288a4eedbe5b4420b42fafe14d47e0e6" is_pre_election: true
I20260928 02:38:51.027721 10719 raft_consensus.cc:2454] T 075a69c6ef664e17abbe483bd2182e9d P 288a4eedbe5b4420b42fafe14d47e0e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d6e89af96a743c58848903c7157c6b7 in term 0.
I20260928 02:38:51.028007 10385 leader_election.cc:302] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288a4eedbe5b4420b42fafe14d47e0e6, 4d6e89af96a743c58848903c7157c6b7; no voters:
I20260928 02:38:51.028003 10585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075a69c6ef664e17abbe483bd2182e9d" candidate_uuid: "4d6e89af96a743c58848903c7157c6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40861cb7107141f5be9fa597880ba515" is_pre_election: true
I20260928 02:38:51.028184 10789 raft_consensus.cc:2781] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:51.028219 10585 raft_consensus.cc:2454] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d6e89af96a743c58848903c7157c6b7 in term 0.
I20260928 02:38:51.028249 10789 raft_consensus.cc:482] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:51.028278 10789 raft_consensus.cc:3037] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:51.028961 10789 raft_consensus.cc:504] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288a4eedbe5b4420b42fafe14d47e0e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 40707 } } peers { permanent_uuid: "4d6e89af96a743c58848903c7157c6b7" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38339 } } peers { permanent_uuid: "40861cb7107141f5be9fa597880ba515" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39037 } }
I20260928 02:38:51.029088 10789 leader_election.cc:288] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [CANDIDATE]: Term 1 election: Requested vote from peers 288a4eedbe5b4420b42fafe14d47e0e6 (127.9.130.3:40707), 40861cb7107141f5be9fa597880ba515 (127.9.130.2:39037)
I20260928 02:38:51.029224 10719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075a69c6ef664e17abbe483bd2182e9d" candidate_uuid: "4d6e89af96a743c58848903c7157c6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288a4eedbe5b4420b42fafe14d47e0e6"
I20260928 02:38:51.029258 10585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075a69c6ef664e17abbe483bd2182e9d" candidate_uuid: "4d6e89af96a743c58848903c7157c6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40861cb7107141f5be9fa597880ba515"
I20260928 02:38:51.029302 10719 raft_consensus.cc:3037] T 075a69c6ef664e17abbe483bd2182e9d P 288a4eedbe5b4420b42fafe14d47e0e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:51.029325 10585 raft_consensus.cc:3037] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:51.029964 10719 raft_consensus.cc:2454] T 075a69c6ef664e17abbe483bd2182e9d P 288a4eedbe5b4420b42fafe14d47e0e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d6e89af96a743c58848903c7157c6b7 in term 1.
I20260928 02:38:51.030035 10585 raft_consensus.cc:2454] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d6e89af96a743c58848903c7157c6b7 in term 1.
I20260928 02:38:51.030221 10385 leader_election.cc:302] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40861cb7107141f5be9fa597880ba515, 4d6e89af96a743c58848903c7157c6b7; no voters:
I20260928 02:38:51.030316 10789 raft_consensus.cc:2781] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:51.030491 10789 raft_consensus.cc:686] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [term 1 LEADER]: Becoming Leader. State: Replica: 4d6e89af96a743c58848903c7157c6b7, State: Running, Role: LEADER
I20260928 02:38:51.030592 10789 consensus_queue.cc:237] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288a4eedbe5b4420b42fafe14d47e0e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 40707 } } peers { permanent_uuid: "4d6e89af96a743c58848903c7157c6b7" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38339 } } peers { permanent_uuid: "40861cb7107141f5be9fa597880ba515" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39037 } }
I20260928 02:38:51.031370 10306 catalog_manager.cc:5495] T 075a69c6ef664e17abbe483bd2182e9d P 4d6e89af96a743c58848903c7157c6b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d6e89af96a743c58848903c7157c6b7 (127.9.130.1). New cstate: current_term: 1 leader_uuid: "4d6e89af96a743c58848903c7157c6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288a4eedbe5b4420b42fafe14d47e0e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d6e89af96a743c58848903c7157c6b7" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40861cb7107141f5be9fa597880ba515" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39037 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:38:51 dist-test-slave-kkrh krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.1@KRBTEST.COM
I20260928 02:38:51.042948 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:51.035274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.9.130.1:38339)
0928 02:38:51.035318 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:51.035320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:51.035419 (+ 99us) server_negotiation.cc:407] Connection header received
0928 02:38:51.035470 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:51.035471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:51.035479 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:51.035563 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:51.036231 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.036708 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.037445 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.037618 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.037670 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:51.040116 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:51.040123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:51.040124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:51.040130 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:51.041961 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:51.042305 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:51.042308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:51.042308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:51.042362 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:51.042529 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:51.042530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:51.042530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:51.042653 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:51.042765 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:51.042822 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 02:38:51.042884 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 02:38:51 dist-test-slave-kkrh krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563131, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260928 02:38:51.062227 10632 tablet.cc:2369] T 075a69c6ef664e17abbe483bd2182e9d P 40861cb7107141f5be9fa597880ba515: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:38:51 dist-test-slave-kkrh krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563131, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.130.1@KRBTEST.COM
I20260928 02:38:51.065726 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:51.058581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.9.130.1:38339)
0928 02:38:51.058664 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:51.058666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:51.058689 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:38:51.058761 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:51.058761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:51.058771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:51.058819 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:51.059317 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.059780 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.060573 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.060774 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.060828 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:51.063153 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:51.063154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:51.063154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:51.063160 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:51.064507 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:51.064833 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:51.064834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:51.064834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:51.064878 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:51.065109 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:51.065109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:51.065109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:51.065191 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:51.065266 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:51.065486 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:38:51.065554 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 02:38:51 dist-test-slave-kkrh krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:38:51 dist-test-slave-kkrh krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.1@KRBTEST.COM
I20260928 02:38:51.087462 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:51.080667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.9.130.1:38339)
0928 02:38:51.080736 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:51.080739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:51.080871 (+ 132us) server_negotiation.cc:407] Connection header received
0928 02:38:51.080939 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:51.080940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:51.080952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:51.081019 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:51.081556 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.082029 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.082780 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:51.082976 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:51.083081 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:51.085217 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:51.085218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:51.085219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:51.085226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:51.086344 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:51.086672 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:51.086673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:51.086673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:51.086729 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:51.086928 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:51.086929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:51.086930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:51.087025 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:51.087147 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:51.087232 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:38:51.087305 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 02:38:51.087939 10411 tablet_service.cc:2908] Rejecting scan request for tablet 075a69c6ef664e17abbe483bd2182e9d: Uninitialized: clean time has not yet been initialized
I20260928 02:38:51.088997 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 10368
I20260928 02:38:51.094290 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 10502
I20260928 02:38:51.098994 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 10636
I20260928 02:38:51.103673 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 10274
2026-09-28T02:38:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1918 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:51 dist-test-slave-kkrh krb5kdc[10806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:51 dist-test-slave-kkrh krb5kdc[10806](info): set up 2 sockets
Sep 28 02:38:51 dist-test-slave-kkrh krb5kdc[10806](info): commencing operation
krb5kdc: starting...
W20260928 02:38:52.215855 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:38:52 dist-test-slave-kkrh krb5kdc[10806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:38:52.332509 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:40663
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:40663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:52.411239 10819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:52.411456 10819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:52.411479 10819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:52.412560 10819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:52.412595 10819 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:38:52.412607 10819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:52.412621 10819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:52.412632 10819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:52.412644 10819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:52.414003 10819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:40663
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:40663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:52.414186 10819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:52.414342 10819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:52.419162 10824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:52.419159 10825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:52.419415 10819 server_base.cc:1034] running on GCE node
W20260928 02:38:52.419183 10827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:52.419736 10819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:52.420038 10819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:52.421177 10819 hybrid_clock.cc:648] HybridClock initialized: now 1790563132421164 us; error 29 us; skew 500 ppm
Sep 28 02:38:52 dist-test-slave-kkrh krb5kdc[10806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:52.423488 10819 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:38:52.423996 10819 webserver.cc:460] Webserver started at http://127.9.130.62:36893/ using document root <none> and password file <none>
I20260928 02:38:52.424216 10819 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:52.424259 10819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:52.424396 10819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:52.425454 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "d1d4d6e5e0ed497e99eb76f7e262743d"
format_stamp: "Formatted at 2026-09-28 02:38:52 on dist-test-slave-kkrh"
server_key: "c22ffc343d0bef514f30a233d163f32e"
server_key_iv: "64d6fb85e64bde446bb5029fb98bf8a5"
server_key_version: "encryptionkey@0"
I20260928 02:38:52.425817 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "d1d4d6e5e0ed497e99eb76f7e262743d"
format_stamp: "Formatted at 2026-09-28 02:38:52 on dist-test-slave-kkrh"
server_key: "c22ffc343d0bef514f30a233d163f32e"
server_key_iv: "64d6fb85e64bde446bb5029fb98bf8a5"
server_key_version: "encryptionkey@0"
I20260928 02:38:52.427093 10819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:52.427793 10834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:52.427982 10819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:52.428078 10819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "d1d4d6e5e0ed497e99eb76f7e262743d"
format_stamp: "Formatted at 2026-09-28 02:38:52 on dist-test-slave-kkrh"
server_key: "c22ffc343d0bef514f30a233d163f32e"
server_key_iv: "64d6fb85e64bde446bb5029fb98bf8a5"
server_key_version: "encryptionkey@0"
I20260928 02:38:52.428164 10819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:52.442345 10819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:52.444301 10819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:52.444471 10819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:52.449131 10819 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:40663
I20260928 02:38:52.449182 10896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:40663 every 8 connection(s)
I20260928 02:38:52.449528 10819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:52.450161 10897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:52.452421 10897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d: Bootstrap starting.
I20260928 02:38:52.453003 10897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:52.453317 10897 log.cc:824] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:52.453975 10897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d: No bootstrap required, opened a new log
I20260928 02:38:52.456066 10897 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d4d6e5e0ed497e99eb76f7e262743d" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 40663 } }
I20260928 02:38:52.456197 10897 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:52.456218 10897 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d4d6e5e0ed497e99eb76f7e262743d, State: Initialized, Role: FOLLOWER
I20260928 02:38:52.456389 10897 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d4d6e5e0ed497e99eb76f7e262743d" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 40663 } }
I20260928 02:38:52.456485 10897 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:52.456539 10897 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:52.456589 10897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:52.456964 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 10819
I20260928 02:38:52.457084 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:38:52.457186 10897 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d4d6e5e0ed497e99eb76f7e262743d" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 40663 } }
I20260928 02:38:52.457280 9736 external_mini_cluster.cc:1428] Setting key e805d61e1721c57b651a8819fb49d904
I20260928 02:38:52.457298 10897 leader_election.cc:302] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1d4d6e5e0ed497e99eb76f7e262743d; no voters:
I20260928 02:38:52.457463 10897 leader_election.cc:288] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:52.457517 10900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:52.457650 10900 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [term 1 LEADER]: Becoming Leader. State: Replica: d1d4d6e5e0ed497e99eb76f7e262743d, State: Running, Role: LEADER
I20260928 02:38:52.457715 10897 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:52.457727 10900 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d4d6e5e0ed497e99eb76f7e262743d" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 40663 } }
I20260928 02:38:52.458061 10900 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1d4d6e5e0ed497e99eb76f7e262743d. Latest consensus state: current_term: 1 leader_uuid: "d1d4d6e5e0ed497e99eb76f7e262743d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d4d6e5e0ed497e99eb76f7e262743d" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 40663 } } }
I20260928 02:38:52.458151 10900 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:52.458069 10901 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1d4d6e5e0ed497e99eb76f7e262743d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d4d6e5e0ed497e99eb76f7e262743d" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 40663 } } }
I20260928 02:38:52.458276 10901 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:52.458452 10908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:52.458974 10908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:52.460566 10908 catalog_manager.cc:1347] Generated new cluster ID: fc3c769ca660404682d3633fa27227fc
I20260928 02:38:52.460616 10908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:38:52 dist-test-slave-kkrh krb5kdc[10806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:52.468261 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.457717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54776 (local address 127.9.130.62:40663)
0928 02:38:52.458343 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:52.458352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:52.458386 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:38:52.458546 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:52.458554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:52.458617 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:52.458775 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:52.459712 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.460981 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.461741 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.462165 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.462303 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.464653 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:52.464657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:52.464657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:52.464668 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:52.466683 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.467068 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.467073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.467074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.467168 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.467379 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.467380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.467380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.467612 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:52.467784 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:52.467901 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 02:38:52.468043 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":67,"threads_started":1}
I20260928 02:38:52.469529 10908 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:52.470109 10908 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:52.475689 10908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d: Generated new TSK 0
I20260928 02:38:52.475869 10908 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:38:52.506402 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:40663
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:52.610291 10923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:52.610489 10923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:52.610513 10923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:52.611729 10923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:52.611768 10923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:52.611788 10923 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:38:52.611804 10923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:38:52.613240 10923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.130.62:40663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:52.613466 10923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:52.613683 10923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:52.619338 10931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:52.619313 10928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:52.619303 10929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:52.619645 10923 server_base.cc:1034] running on GCE node
I20260928 02:38:52.619861 10923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:52.620144 10923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:52.621304 10923 hybrid_clock.cc:648] HybridClock initialized: now 1790563132621291 us; error 29 us; skew 500 ppm
Sep 28 02:38:52 dist-test-slave-kkrh krb5kdc[10806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:52.623880 10923 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:38:52.624400 10923 webserver.cc:460] Webserver started at http://127.9.130.1:46097/ using document root <none> and password file <none>
I20260928 02:38:52.624642 10923 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:52.624711 10923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:52.624845 10923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:52.625909 10923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "57a6bfe4c06b4dd7a02cae25066042f0"
format_stamp: "Formatted at 2026-09-28 02:38:52 on dist-test-slave-kkrh"
server_key: "427a6275c325db321922a21aa2c40186"
server_key_iv: "02d03fd28a9308703a49e62ee3769655"
server_key_version: "encryptionkey@0"
I20260928 02:38:52.626274 10923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "57a6bfe4c06b4dd7a02cae25066042f0"
format_stamp: "Formatted at 2026-09-28 02:38:52 on dist-test-slave-kkrh"
server_key: "427a6275c325db321922a21aa2c40186"
server_key_iv: "02d03fd28a9308703a49e62ee3769655"
server_key_version: "encryptionkey@0"
I20260928 02:38:52.627596 10923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:52.628391 10938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:52.628594 10923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:52.628686 10923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "57a6bfe4c06b4dd7a02cae25066042f0"
format_stamp: "Formatted at 2026-09-28 02:38:52 on dist-test-slave-kkrh"
server_key: "427a6275c325db321922a21aa2c40186"
server_key_iv: "02d03fd28a9308703a49e62ee3769655"
server_key_version: "encryptionkey@0"
I20260928 02:38:52.628772 10923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:52.642311 10923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:52.644701 10923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:52.644866 10923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:52.645340 10923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:52.645397 10923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:52.645437 10923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:52.645493 10923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:52.652216 10923 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:46599
I20260928 02:38:52.652302 11053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:46599 every 8 connection(s)
Sep 28 02:38:52 dist-test-slave-kkrh krb5kdc[10806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:52.652637 10923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:38:52.655361 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.645882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:39747 (local address 127.9.130.62:40663)
0928 02:38:52.645970 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:52.645972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:52.646181 (+ 209us) server_negotiation.cc:407] Connection header received
0928 02:38:52.646677 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:52.646679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:52.646697 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:52.646781 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:52.647597 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.648480 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.650033 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.650361 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.650448 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.653041 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:52.653042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:52.653043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:52.653051 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:52.654268 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.654631 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.654632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.654633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.654683 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.654881 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.654882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.654882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.654990 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:52.655041 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:52.655241 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 02:38:52.655301 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:38:52.655541 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.645857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:40663 (local address 127.9.130.1:39747)
0928 02:38:52.646038 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:52.646076 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:52.646572 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:52.646843 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:52.646847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:52.647215 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:52.647487 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:52.647494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.648582 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.648583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:52.649938 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:52.649941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.650474 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.650475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:52.650570 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.651103 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:52.651115 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:52.652923 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:52.654339 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:52.654344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:52.654346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:52.654564 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:52.654740 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:52.654741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:52.654741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:52.654829 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:52.655046 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:52.655049 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:52.655151 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 02:38:52.655230 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:38:52.655337 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260928 02:38:52.656111 11054 heartbeater.cc:347] Connected to a master server at 127.9.130.62:40663
I20260928 02:38:52.656190 11054 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:52.656360 11054 heartbeater.cc:511] Master 127.9.130.62:40663 requested a full tablet report, sending...
I20260928 02:38:52.656904 10850 ts_manager.cc:194] Registered new tserver with Master: 57a6bfe4c06b4dd7a02cae25066042f0 (127.9.130.1:46599)
I20260928 02:38:52.657580 10850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:39747
I20260928 02:38:52.661744 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 10923
I20260928 02:38:52.661840 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:38:52.662022 9736 external_mini_cluster.cc:1428] Setting key 6850485fe90ff1183308883088ee2bac
I20260928 02:38:52.662657 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.657090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:40663 (local address 127.9.130.1:59803)
0928 02:38:52.657223 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:52.657232 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:52.657274 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:52.657870 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:52.657871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:52.658075 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:52.658131 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:52.658133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.658744 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.658745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:52.659320 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:52.659322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.659640 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.659641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:52.659691 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.660060 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:52.660064 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:52.660321 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:52.661651 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:52.661652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:52.661653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:52.661873 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:52.662072 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:52.662073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:52.662074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:52.662141 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:52.662421 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:52.662422 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:52.662466 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:38:52.662521 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:38:52.662592 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260928 02:38:52.662711 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.657127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:59803 (local address 127.9.130.62:40663)
0928 02:38:52.657766 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:52.657767 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:52.657775 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:52.657789 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:52.657789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:52.657796 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:52.657865 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:52.658209 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.658647 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.659385 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.659580 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.659633 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.660402 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:52.660403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:52.660404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:52.660434 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:52.661573 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.661956 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.661956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.661957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.662007 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.662209 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.662210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.662210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.662362 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:52.662424 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:52.662562 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 02:38:52.662634 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625}
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:38:52.687239 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:40663
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:52.766273 11063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:52.766475 11063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:52.766498 11063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:52.767670 11063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:52.767714 11063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:52.767733 11063 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:38:52.767751 11063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:38:52.769222 11063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.130.62:40663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:52.769457 11063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:52.769693 11063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:38:52.774853 11063 server_base.cc:1034] running on GCE node
W20260928 02:38:52.774892 11068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:52.774971 11069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:52.774899 11071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:52.775220 11063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:52.775520 11063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:52.776667 11063 hybrid_clock.cc:648] HybridClock initialized: now 1790563132776602 us; error 75 us; skew 500 ppm
Sep 28 02:38:52 dist-test-slave-kkrh krb5kdc[10806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:52.779067 11063 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:38:52.779506 11063 webserver.cc:460] Webserver started at http://127.9.130.2:45695/ using document root <none> and password file <none>
I20260928 02:38:52.779685 11063 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:52.779722 11063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:52.779806 11063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:52.780861 11063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "5389e19e864741a0ae58f47ad32297fa"
format_stamp: "Formatted at 2026-09-28 02:38:52 on dist-test-slave-kkrh"
server_key: "d90c589afc5625e3eee3b06871451fe9"
server_key_iv: "b76b2278a792d325fc52806c6390e56b"
server_key_version: "encryptionkey@0"
I20260928 02:38:52.781159 11063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "5389e19e864741a0ae58f47ad32297fa"
format_stamp: "Formatted at 2026-09-28 02:38:52 on dist-test-slave-kkrh"
server_key: "d90c589afc5625e3eee3b06871451fe9"
server_key_iv: "b76b2278a792d325fc52806c6390e56b"
server_key_version: "encryptionkey@0"
I20260928 02:38:52.782435 11063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:38:52.783208 11078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:52.783383 11063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:38:52.783519 11063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "5389e19e864741a0ae58f47ad32297fa"
format_stamp: "Formatted at 2026-09-28 02:38:52 on dist-test-slave-kkrh"
server_key: "d90c589afc5625e3eee3b06871451fe9"
server_key_iv: "b76b2278a792d325fc52806c6390e56b"
server_key_version: "encryptionkey@0"
I20260928 02:38:52.783596 11063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:52.790654 11063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:52.792584 11063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:52.792738 11063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:52.793313 11063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:52.793360 11063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:52.793393 11063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:52.793455 11063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:38:52 dist-test-slave-kkrh krb5kdc[10806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:52.800261 11063 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:34219
I20260928 02:38:52.800462 11193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:34219 every 8 connection(s)
I20260928 02:38:52.800673 11063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:38:52.801582 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 11063
I20260928 02:38:52.801697 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:38:52.801880 9736 external_mini_cluster.cc:1428] Setting key f32672b0d67c0fc9c4c99a425b6f35c3
I20260928 02:38:52.803680 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.793699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:58053 (local address 127.9.130.62:40663)
0928 02:38:52.793749 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:52.793753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:52.793954 (+ 201us) server_negotiation.cc:407] Connection header received
0928 02:38:52.794460 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:52.794461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:52.794473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:52.794566 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:52.795395 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.796053 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.797086 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.797297 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.797381 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.800274 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:52.800275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:52.800276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:52.800284 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:52.801760 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.802393 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.802394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.802394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.802480 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.802779 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.802780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.802781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.802991 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:52.803076 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:52.803514 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 02:38:52.803599 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:38:52.803995 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.793687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:40663 (local address 127.9.130.2:58053)
0928 02:38:52.793823 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:52.793861 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:52.794356 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:52.794638 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:52.794642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:52.795020 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:52.795309 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:52.795316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.796152 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.796153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:52.796958 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:52.796960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.797389 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.797390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:52.797512 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.798018 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:52.798026 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:52.800159 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:52.801842 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:52.801849 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:52.801852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:52.802151 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:52.802560 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:52.802561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:52.802561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:52.802708 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:52.803074 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:52.803080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:52.803327 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 02:38:52.803499 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 02:38:52.803694 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":78,"threads_started":1}
I20260928 02:38:52.804437 11194 heartbeater.cc:347] Connected to a master server at 127.9.130.62:40663
I20260928 02:38:52.804525 11194 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:52.804701 11194 heartbeater.cc:511] Master 127.9.130.62:40663 requested a full tablet report, sending...
I20260928 02:38:52.805068 10850 ts_manager.cc:194] Registered new tserver with Master: 5389e19e864741a0ae58f47ad32297fa (127.9.130.2:34219)
I20260928 02:38:52.805505 10850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:58053
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
I20260928 02:38:52.811221 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.805552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:55619 (local address 127.9.130.62:40663)
0928 02:38:52.805591 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:52.805593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:52.805702 (+ 109us) server_negotiation.cc:407] Connection header received
0928 02:38:52.805771 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:52.805772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:52.805782 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:52.805825 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:52.806250 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.806688 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.807428 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.807626 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.807680 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.808705 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:52.808706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:52.808707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:52.808712 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:52.810095 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.810490 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.810490 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.810491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.810536 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.810731 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.810732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.810732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.810850 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:52.810900 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:52.811097 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:38:52.811166 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:38:52.811290 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.805477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:40663 (local address 127.9.130.2:55619)
0928 02:38:52.805645 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:52.805657 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:52.805712 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:52.805878 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:52.805880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:52.806104 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:52.806170 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:52.806172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.806791 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.806792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:52.807321 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:52.807323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.807704 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.807706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:52.807795 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.808192 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:52.808198 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:52.808616 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:52.810180 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:52.810181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:52.810182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:52.810396 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:52.810605 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:52.810606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:52.810606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:52.810668 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:52.810915 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:52.810916 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:52.810966 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 02:38:52.811048 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:38:52.811152 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:38:52.826712 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:40663
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:52.905903 11203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:52.906133 11203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:52.906172 11203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:52.907322 11203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:52.907388 11203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:52.907425 11203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:38:52.907464 11203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:38:52.908885 11203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.130.62:40663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:52.909152 11203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:52.909370 11203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:52.914430 11208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:52.914453 11209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:52.914508 11203 server_base.cc:1034] running on GCE node
W20260928 02:38:52.914517 11211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:52.914911 11203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:52.915151 11203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:52.916311 11203 hybrid_clock.cc:648] HybridClock initialized: now 1790563132916296 us; error 53 us; skew 500 ppm
Sep 28 02:38:52 dist-test-slave-kkrh krb5kdc[10806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:52.918651 11203 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:38:52.919015 11203 webserver.cc:460] Webserver started at http://127.9.130.3:35703/ using document root <none> and password file <none>
I20260928 02:38:52.919190 11203 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:52.919229 11203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:52.919313 11203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:52.920382 11203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "e04e8766f8b648fd89b3b0463f888928"
format_stamp: "Formatted at 2026-09-28 02:38:52 on dist-test-slave-kkrh"
server_key: "4383c0c6a829165705cad5da176b7fd1"
server_key_iv: "72774f6a374cb43298eff5e48adb42df"
server_key_version: "encryptionkey@0"
I20260928 02:38:52.920687 11203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "e04e8766f8b648fd89b3b0463f888928"
format_stamp: "Formatted at 2026-09-28 02:38:52 on dist-test-slave-kkrh"
server_key: "4383c0c6a829165705cad5da176b7fd1"
server_key_iv: "72774f6a374cb43298eff5e48adb42df"
server_key_version: "encryptionkey@0"
I20260928 02:38:52.921912 11203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:38:52.922581 11218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:52.922770 11203 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:52.922850 11203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "e04e8766f8b648fd89b3b0463f888928"
format_stamp: "Formatted at 2026-09-28 02:38:52 on dist-test-slave-kkrh"
server_key: "4383c0c6a829165705cad5da176b7fd1"
server_key_iv: "72774f6a374cb43298eff5e48adb42df"
server_key_version: "encryptionkey@0"
I20260928 02:38:52.922904 11203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:52.953313 11203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:52.955668 11203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:52.955780 11203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:52.956393 11203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:52.956442 11203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:52.956465 11203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:52.956477 11203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:38:52 dist-test-slave-kkrh krb5kdc[10806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:52.963593 11203 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:39563
I20260928 02:38:52.963658 11333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:39563 every 8 connection(s)
I20260928 02:38:52.964026 11203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:38:52.966773 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.956867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:53343 (local address 127.9.130.62:40663)
0928 02:38:52.956937 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:52.956940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:52.957130 (+ 190us) server_negotiation.cc:407] Connection header received
0928 02:38:52.957753 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:52.957754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:52.957768 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:52.957836 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:52.958642 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.959498 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.960948 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.961306 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.961380 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.964174 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:52.964176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:52.964176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:52.964183 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:52.965685 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.966042 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.966043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.966043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.966090 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.966273 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.966274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.966274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.966372 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:52.966421 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:52.966653 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 02:38:52.966711 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:38:52.966926 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.956835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:40663 (local address 127.9.130.3:53343)
0928 02:38:52.956982 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:52.957022 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:52.957638 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:52.957898 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:52.957902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:52.958246 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:52.958535 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:52.958543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.959608 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.959609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:52.960839 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:52.960841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.961424 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.961424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:52.961584 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.962149 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:52.962157 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:52.964086 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:52.965763 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:52.965768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:52.965770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:52.965976 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:52.966143 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:52.966143 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:52.966143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:52.966224 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:52.966426 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:52.966430 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:52.966544 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 02:38:52.966624 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:38:52.966737 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":97,"threads_started":1}
I20260928 02:38:52.967396 11334 heartbeater.cc:347] Connected to a master server at 127.9.130.62:40663
I20260928 02:38:52.967482 11334 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:52.967664 11334 heartbeater.cc:511] Master 127.9.130.62:40663 requested a full tablet report, sending...
I20260928 02:38:52.968102 10851 ts_manager.cc:194] Registered new tserver with Master: e04e8766f8b648fd89b3b0463f888928 (127.9.130.3:39563)
I20260928 02:38:52.968641 10851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:53343
I20260928 02:38:52.971299 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 11203
I20260928 02:38:52.971387 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:38:52.971573 9736 external_mini_cluster.cc:1428] Setting key 69a9eaec82033c7d2fe0fff03d4155fb
I20260928 02:38:52.972440 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:38:52.974584 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.968481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:40663 (local address 127.9.130.3:33799)
0928 02:38:52.968589 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:52.968600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:52.968675 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:52.969301 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:52.969302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:52.969531 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:52.969608 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:52.969610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.970301 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.970302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:52.970804 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:52.970806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.971167 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.971168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:52.971251 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.971810 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:52.971837 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:52.972137 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:52.973575 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:52.973576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:52.973576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:52.973809 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:52.973995 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:52.973996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:52.973996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:52.974067 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:52.974336 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:52.974338 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:52.974383 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:38:52.974437 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:38:52.974519 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":63,"threads_started":1}
I20260928 02:38:52.974673 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.968538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:33799 (local address 127.9.130.62:40663)
0928 02:38:52.968586 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:52.968588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:52.968649 (+ 61us) server_negotiation.cc:407] Connection header received
0928 02:38:52.968792 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:52.968793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:52.968805 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:52.969284 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:52.969722 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.970209 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.970875 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.971059 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.971138 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.972195 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:52.972196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:52.972196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:52.972203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:52.973492 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.973884 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.973885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.973885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.973932 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.974132 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.974132 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.974132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.974271 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:52.974338 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:52.974481 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 02:38:52.974561 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 02:38:52 dist-test-slave-kkrh krb5kdc[10806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563132, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:38:52 dist-test-slave-kkrh krb5kdc[10806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563132, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:52.994822 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:52.988146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.9.130.62:40663)
0928 02:38:52.988214 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:52.988217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:52.988238 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:38:52.988270 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:52.988271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:52.988280 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:52.988340 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:52.988892 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.989372 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.990123 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:52.990311 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:52.990384 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:52.992569 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:52.992570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:52.992571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:52.992577 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:52.993943 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.994226 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.994227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.994227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.994276 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:52.994414 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:52.994414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:52.994414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:52.994538 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:52.994612 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:52.994699 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:38:52.994756 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:38:52.996444 10851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:38:52.997982 10851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:53.004199 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.000318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46599 (local address 127.0.0.1:59322)
0928 02:38:53.000497 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:53.000528 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:53.000586 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:53.000787 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:53.000789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:53.000792 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:53.000863 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.000865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.001823 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.001824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.002855 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.002858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.003876 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.003877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.003980 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.003996 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 02:38:53.004067 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:38:53.004116 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":25,"thread_start_us":62,"threads_started":1}
I20260928 02:38:53.004240 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.000423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.9.130.1:46599)
0928 02:38:53.000614 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:53.000620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:53.000630 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:38:53.000670 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:53.000671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:53.000686 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:53.000785 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:53.000949 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.001734 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.003032 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.003753 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.003852 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.003914 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:53.004096 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 02:38:53.004138 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260928 02:38:53.004555 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.000318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:39563 (local address 127.0.0.1:48580)
0928 02:38:53.000483 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:53.000522 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:53.000585 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:53.001029 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:53.001030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:53.001032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:53.001084 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.001086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.002023 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.002024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.003198 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.003201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.004208 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.004209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.004276 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.004280 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:53.004404 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 02:38:53.004450 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260928 02:38:53.004607 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.000679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.9.130.3:39563)
0928 02:38:53.000835 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:53.000840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:53.000850 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:38:53.000889 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:53.000890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:53.000906 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:53.001165 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:53.001222 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.001920 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.003286 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.004036 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.004121 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.004172 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:53.004458 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 02:38:53.004504 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260928 02:38:53.005301 11268 tablet_service.cc:1467] Processing CreateTablet for tablet 771a2784cfd443a192836e3c4594bf9b (DEFAULT_TABLE table=test-table [id=0e28ce1e567d4eebbb848c9da6338ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:53.005440 10988 tablet_service.cc:1467] Processing CreateTablet for tablet 771a2784cfd443a192836e3c4594bf9b (DEFAULT_TABLE table=test-table [id=0e28ce1e567d4eebbb848c9da6338ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:53.005682 11268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771a2784cfd443a192836e3c4594bf9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:53.005702 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.000554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:34219 (local address 127.0.0.1:37608)
0928 02:38:53.001111 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:53.001121 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:53.001161 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:53.002271 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:53.002272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:53.002275 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:53.002309 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.002310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.003466 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.003467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.004484 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.004486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.005454 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.005456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.005518 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.005522 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:53.005603 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:38:53.005641 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":65,"threads_started":1}
I20260928 02:38:53.005779 10988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771a2784cfd443a192836e3c4594bf9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:53.005836 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.000944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37608 (local address 127.9.130.2:34219)
0928 02:38:53.002101 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:53.002106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:53.002114 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:53.002155 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:53.002158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:53.002174 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:53.002251 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:53.002373 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.003377 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.004573 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.005331 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.005408 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.005501 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:53.005615 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 02:38:53.005654 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1118,"thread_start_us":84,"threads_started":1}
I20260928 02:38:53.006444 11128 tablet_service.cc:1467] Processing CreateTablet for tablet 771a2784cfd443a192836e3c4594bf9b (DEFAULT_TABLE table=test-table [id=0e28ce1e567d4eebbb848c9da6338ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:53.006750 11128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771a2784cfd443a192836e3c4594bf9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:53.008692 11357 tablet_bootstrap.cc:492] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0: Bootstrap starting.
I20260928 02:38:53.009305 11357 tablet_bootstrap.cc:654] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:53.009617 11357 log.cc:824] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:53.010021 11360 tablet_bootstrap.cc:492] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa: Bootstrap starting.
I20260928 02:38:53.010089 11358 tablet_bootstrap.cc:492] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928: Bootstrap starting.
I20260928 02:38:53.010545 11360 tablet_bootstrap.cc:654] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:53.010483 11357 tablet_bootstrap.cc:492] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0: No bootstrap required, opened a new log
I20260928 02:38:53.010636 11357 ts_tablet_manager.cc:1397] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:53.010881 11360 log.cc:824] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:53.011018 11358 tablet_bootstrap.cc:654] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:53.011332 11358 log.cc:824] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:53.012061 11358 tablet_bootstrap.cc:492] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928: No bootstrap required, opened a new log
I20260928 02:38:53.012132 11358 ts_tablet_manager.cc:1397] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:38:53.012346 11360 tablet_bootstrap.cc:492] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa: No bootstrap required, opened a new log
I20260928 02:38:53.012422 11360 ts_tablet_manager.cc:1397] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:53.012756 11357 raft_consensus.cc:348] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.012929 11357 raft_consensus.cc:374] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:53.012964 11357 raft_consensus.cc:729] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a6bfe4c06b4dd7a02cae25066042f0, State: Initialized, Role: FOLLOWER
I20260928 02:38:53.013116 11357 consensus_queue.cc:260] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.013376 11357 ts_tablet_manager.cc:1428] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:38:53.013509 11054 heartbeater.cc:503] Master 127.9.130.62:40663 was elected leader, sending a full tablet report...
I20260928 02:38:53.014211 11360 raft_consensus.cc:348] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.014361 11360 raft_consensus.cc:374] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:53.014410 11360 raft_consensus.cc:729] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5389e19e864741a0ae58f47ad32297fa, State: Initialized, Role: FOLLOWER
I20260928 02:38:53.014519 11360 consensus_queue.cc:260] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.014492 11358 raft_consensus.cc:348] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.014644 11358 raft_consensus.cc:374] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:53.014683 11358 raft_consensus.cc:729] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04e8766f8b648fd89b3b0463f888928, State: Initialized, Role: FOLLOWER
I20260928 02:38:53.014771 11360 ts_tablet_manager.cc:1428] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:38:53.014832 11358 consensus_queue.cc:260] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.014895 11194 heartbeater.cc:503] Master 127.9.130.62:40663 was elected leader, sending a full tablet report...
I20260928 02:38:53.015110 11358 ts_tablet_manager.cc:1428] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:38:53.015118 11334 heartbeater.cc:503] Master 127.9.130.62:40663 was elected leader, sending a full tablet report...
I20260928 02:38:53.019088 11365 raft_consensus.cc:482] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:53.019167 11365 raft_consensus.cc:504] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.019390 11365 leader_election.cc:288] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5389e19e864741a0ae58f47ad32297fa (127.9.130.2:34219), 57a6bfe4c06b4dd7a02cae25066042f0 (127.9.130.1:46599)
I20260928 02:38:53.023111 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.019615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:48693 (local address 127.9.130.1:46599)
0928 02:38:53.019664 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:53.019666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:53.019766 (+ 100us) server_negotiation.cc:407] Connection header received
0928 02:38:53.019794 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:53.019794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:53.019805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:53.019874 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:53.020050 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.020760 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.022231 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.022755 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.022830 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.022840 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:53.023014 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 02:38:53.023059 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:38:53.023110 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.019704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:44373 (local address 127.9.130.2:34219)
0928 02:38:53.019742 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:53.019743 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:53.019752 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:53.019797 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:53.019798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:53.019806 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:53.019874 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:53.020050 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.020788 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.022231 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.022777 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.022837 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.022844 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:53.023014 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 02:38:53.023049 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:38:53.023132 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.019551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46599 (local address 127.9.130.3:48693)
0928 02:38:53.019669 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:53.019681 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:53.019744 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:53.019887 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:53.019888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:53.019889 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:53.019973 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.019975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.020851 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.020852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.022143 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.022145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.022845 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.022846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.022935 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.022936 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:38:53.022993 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:38:53.023052 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":62}
I20260928 02:38:53.023137 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.019551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:34219 (local address 127.9.130.3:44373)
0928 02:38:53.019694 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:53.019702 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:53.019744 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:53.019887 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:53.019888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:53.019889 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:53.019984 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.019985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.020853 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.020853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.022143 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.022145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.022845 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.022846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.022932 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.022936 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:53.022993 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:38:53.023037 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":132,"thread_start_us":77,"threads_started":1}
I20260928 02:38:53.023487 11008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a2784cfd443a192836e3c4594bf9b" candidate_uuid: "e04e8766f8b648fd89b3b0463f888928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" is_pre_election: true
I20260928 02:38:53.023522 11148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a2784cfd443a192836e3c4594bf9b" candidate_uuid: "e04e8766f8b648fd89b3b0463f888928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5389e19e864741a0ae58f47ad32297fa" is_pre_election: true
I20260928 02:38:53.023614 11008 raft_consensus.cc:2454] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e04e8766f8b648fd89b3b0463f888928 in term 0.
I20260928 02:38:53.023614 11148 raft_consensus.cc:2454] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e04e8766f8b648fd89b3b0463f888928 in term 0.
I20260928 02:38:53.023808 11219 leader_election.cc:302] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5389e19e864741a0ae58f47ad32297fa, e04e8766f8b648fd89b3b0463f888928; no voters:
I20260928 02:38:53.023929 11365 raft_consensus.cc:2781] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:53.023978 11365 raft_consensus.cc:482] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:53.024008 11365 raft_consensus.cc:3037] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:53.025238 11365 raft_consensus.cc:504] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.025403 11365 leader_election.cc:288] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [CANDIDATE]: Term 1 election: Requested vote from peers 5389e19e864741a0ae58f47ad32297fa (127.9.130.2:34219), 57a6bfe4c06b4dd7a02cae25066042f0 (127.9.130.1:46599)
I20260928 02:38:53.025512 11148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a2784cfd443a192836e3c4594bf9b" candidate_uuid: "e04e8766f8b648fd89b3b0463f888928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5389e19e864741a0ae58f47ad32297fa"
I20260928 02:38:53.025545 11008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a2784cfd443a192836e3c4594bf9b" candidate_uuid: "e04e8766f8b648fd89b3b0463f888928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a6bfe4c06b4dd7a02cae25066042f0"
I20260928 02:38:53.025588 11148 raft_consensus.cc:3037] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:53.025614 11008 raft_consensus.cc:3037] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:53.026253 11148 raft_consensus.cc:2454] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e04e8766f8b648fd89b3b0463f888928 in term 1.
I20260928 02:38:53.026304 11008 raft_consensus.cc:2454] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e04e8766f8b648fd89b3b0463f888928 in term 1.
I20260928 02:38:53.026435 11219 leader_election.cc:302] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5389e19e864741a0ae58f47ad32297fa, e04e8766f8b648fd89b3b0463f888928; no voters:
I20260928 02:38:53.026552 11365 raft_consensus.cc:2781] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:53.026697 11365 raft_consensus.cc:686] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [term 1 LEADER]: Becoming Leader. State: Replica: e04e8766f8b648fd89b3b0463f888928, State: Running, Role: LEADER
I20260928 02:38:53.026782 11365 consensus_queue.cc:237] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.027537 10850 catalog_manager.cc:5495] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 reported cstate change: term changed from 0 to 1, leader changed from <none> to e04e8766f8b648fd89b3b0463f888928 (127.9.130.3). New cstate: current_term: 1 leader_uuid: "e04e8766f8b648fd89b3b0463f888928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } health_report { overall_health: UNKNOWN } } }
I20260928 02:38:53.035167 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.032670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.9.130.3:39563)
0928 02:38:53.032714 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:53.032716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:53.032725 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:53.032838 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:53.032839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:53.032855 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:53.032933 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:53.033075 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.033523 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.034300 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.034515 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.034599 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.034814 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:53.034905 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:53.034993 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:53.035050 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 02:38:53.035070 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:38:53.037129 11008 raft_consensus.cc:1261] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0 [term 1 FOLLOWER]: Refusing update from remote peer e04e8766f8b648fd89b3b0463f888928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:53.037448 11148 raft_consensus.cc:1261] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa [term 1 FOLLOWER]: Refusing update from remote peer e04e8766f8b648fd89b3b0463f888928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:53.037556 11367 consensus_queue.cc:1035] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:53.037753 11365 consensus_queue.cc:1035] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:53.040409 11371 mvcc.cc:205] Tried to move back new op lower bound from 7334146592917602304 to 7334146592878374912. Current Snapshot: MvccSnapshot[applied={T|T < 7334146592917602304 or (T in {7334146592917602304})}]
I20260928 02:38:53.040558 11372 mvcc.cc:205] Tried to move back new op lower bound from 7334146592917602304 to 7334146592878374912. Current Snapshot: MvccSnapshot[applied={T|T < 7334146592917602304}]
I20260928 02:38:53.043618 10851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:53.043705 10851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:53.044500 10851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d: Sending DeleteTablet for 3 replicas of tablet 771a2784cfd443a192836e3c4594bf9b
I20260928 02:38:53.044842 11128 tablet_service.cc:1514] Processing DeleteTablet for tablet 771a2784cfd443a192836e3c4594bf9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:53 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:37608
I20260928 02:38:53.044829 10988 tablet_service.cc:1514] Processing DeleteTablet for tablet 771a2784cfd443a192836e3c4594bf9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:53 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:59322
I20260928 02:38:53.045051 11380 tablet_replica.cc:330] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa: stopping tablet replica
I20260928 02:38:53.045151 11380 raft_consensus.cc:2229] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:53.045169 11268 tablet_service.cc:1514] Processing DeleteTablet for tablet 771a2784cfd443a192836e3c4594bf9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:53 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:48580
I20260928 02:38:53.045369 11382 tablet_replica.cc:330] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928: stopping tablet replica
I20260928 02:38:53.045495 11382 raft_consensus.cc:2229] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:38:53.045516 11380 raft_consensus.cc:2258] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:53.045605 10851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:38:53.045667 11382 raft_consensus.cc:2258] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 02:38:53.045753 10851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:53.046070 11382 ts_tablet_manager.cc:1905] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:53.046868 10988 tablet_service.cc:1467] Processing CreateTablet for tablet e95d5581947c42db982f75baa92e4810 (DEFAULT_TABLE table=test-table [id=9ee818c50045465481e25425e83529f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:53.046916 11128 tablet_service.cc:1467] Processing CreateTablet for tablet e95d5581947c42db982f75baa92e4810 (DEFAULT_TABLE table=test-table [id=9ee818c50045465481e25425e83529f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:53.046984 10988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95d5581947c42db982f75baa92e4810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:53.047011 11128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95d5581947c42db982f75baa92e4810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:53.047690 11382 ts_tablet_manager.cc:1918] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:53.047750 11382 log.cc:1196] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/771a2784cfd443a192836e3c4594bf9b
I20260928 02:38:53.048022 11382 ts_tablet_manager.cc:1939] T 771a2784cfd443a192836e3c4594bf9b P e04e8766f8b648fd89b3b0463f888928: Deleting consensus metadata
I20260928 02:38:53.048285 11360 tablet_bootstrap.cc:492] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa: Bootstrap starting.
I20260928 02:38:53.048388 11357 tablet_bootstrap.cc:492] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0: Bootstrap starting.
I20260928 02:38:53.048520 10835 catalog_manager.cc:4841] TS e04e8766f8b648fd89b3b0463f888928 (127.9.130.3:39563): tablet 771a2784cfd443a192836e3c4594bf9b (table test-table [id=0e28ce1e567d4eebbb848c9da6338ed8]) successfully deleted
I20260928 02:38:53.048568 11268 tablet_service.cc:1467] Processing CreateTablet for tablet e95d5581947c42db982f75baa92e4810 (DEFAULT_TABLE table=test-table [id=9ee818c50045465481e25425e83529f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:53.048663 11268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95d5581947c42db982f75baa92e4810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:53.048705 11360 tablet_bootstrap.cc:654] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:53.049219 11360 tablet_bootstrap.cc:492] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa: No bootstrap required, opened a new log
I20260928 02:38:53.049250 11360 ts_tablet_manager.cc:1397] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:53.049353 11360 raft_consensus.cc:348] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } }
I20260928 02:38:53.049409 11360 raft_consensus.cc:374] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:53.049423 11360 raft_consensus.cc:729] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5389e19e864741a0ae58f47ad32297fa, State: Initialized, Role: FOLLOWER
I20260928 02:38:53.049455 11357 tablet_bootstrap.cc:654] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:53.049491 11360 consensus_queue.cc:260] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } }
I20260928 02:38:53.049602 11360 ts_tablet_manager.cc:1428] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:53.050328 11358 tablet_bootstrap.cc:492] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928: Bootstrap starting.
I20260928 02:38:53.050387 11357 tablet_bootstrap.cc:492] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0: No bootstrap required, opened a new log
I20260928 02:38:53.050427 11357 ts_tablet_manager.cc:1397] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:53.050585 11357 raft_consensus.cc:348] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } }
I20260928 02:38:53.050657 11357 raft_consensus.cc:374] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:53.050676 11357 raft_consensus.cc:729] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a6bfe4c06b4dd7a02cae25066042f0, State: Initialized, Role: FOLLOWER
I20260928 02:38:53.050730 11358 tablet_bootstrap.cc:654] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:53.050720 11357 consensus_queue.cc:260] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } }
I20260928 02:38:53.050804 11357 ts_tablet_manager.cc:1428] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:53.050964 11380 ts_tablet_manager.cc:1905] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:38:53.051229 11195 tablet.cc:2369] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:38:53.051709 11358 tablet_bootstrap.cc:492] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928: No bootstrap required, opened a new log
I20260928 02:38:53.051762 11358 ts_tablet_manager.cc:1397] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:53.051934 11358 raft_consensus.cc:348] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } }
I20260928 02:38:53.052007 11358 raft_consensus.cc:374] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:53.052028 11358 raft_consensus.cc:729] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04e8766f8b648fd89b3b0463f888928, State: Initialized, Role: FOLLOWER
I20260928 02:38:53.052071 11358 consensus_queue.cc:260] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } }
I20260928 02:38:53.052161 11358 ts_tablet_manager.cc:1428] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:53.052539 11380 ts_tablet_manager.cc:1918] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:53.052608 11380 log.cc:1196] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/771a2784cfd443a192836e3c4594bf9b
I20260928 02:38:53.052886 11380 ts_tablet_manager.cc:1939] T 771a2784cfd443a192836e3c4594bf9b P 5389e19e864741a0ae58f47ad32297fa: Deleting consensus metadata
I20260928 02:38:53.053224 11381 tablet_replica.cc:330] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0: stopping tablet replica
I20260928 02:38:53.053287 10835 catalog_manager.cc:4841] TS 5389e19e864741a0ae58f47ad32297fa (127.9.130.2:34219): tablet 771a2784cfd443a192836e3c4594bf9b (table test-table [id=0e28ce1e567d4eebbb848c9da6338ed8]) successfully deleted
I20260928 02:38:53.053355 11381 raft_consensus.cc:2229] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:53.053458 11381 raft_consensus.cc:2258] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:53.053819 11381 ts_tablet_manager.cc:1905] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:53.055131 11381 ts_tablet_manager.cc:1918] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:53.055186 11381 log.cc:1196] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/wal/wals/771a2784cfd443a192836e3c4594bf9b
I20260928 02:38:53.055434 11381 ts_tablet_manager.cc:1939] T 771a2784cfd443a192836e3c4594bf9b P 57a6bfe4c06b4dd7a02cae25066042f0: Deleting consensus metadata
I20260928 02:38:53.055795 10838 catalog_manager.cc:4841] TS 57a6bfe4c06b4dd7a02cae25066042f0 (127.9.130.1:46599): tablet 771a2784cfd443a192836e3c4594bf9b (table test-table [id=0e28ce1e567d4eebbb848c9da6338ed8]) successfully deleted
I20260928 02:38:53.143594 11377 raft_consensus.cc:482] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:53.143702 11377 raft_consensus.cc:504] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } }
I20260928 02:38:53.143894 11377 leader_election.cc:288] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57a6bfe4c06b4dd7a02cae25066042f0 (127.9.130.1:46599), 5389e19e864741a0ae58f47ad32297fa (127.9.130.2:34219)
I20260928 02:38:53.144109 11008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95d5581947c42db982f75baa92e4810" candidate_uuid: "e04e8766f8b648fd89b3b0463f888928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" is_pre_election: true
I20260928 02:38:53.144100 11148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95d5581947c42db982f75baa92e4810" candidate_uuid: "e04e8766f8b648fd89b3b0463f888928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5389e19e864741a0ae58f47ad32297fa" is_pre_election: true
I20260928 02:38:53.144208 11008 raft_consensus.cc:2454] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e04e8766f8b648fd89b3b0463f888928 in term 0.
I20260928 02:38:53.144228 11148 raft_consensus.cc:2454] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e04e8766f8b648fd89b3b0463f888928 in term 0.
I20260928 02:38:53.144410 11219 leader_election.cc:302] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5389e19e864741a0ae58f47ad32297fa, e04e8766f8b648fd89b3b0463f888928; no voters:
I20260928 02:38:53.144512 11377 raft_consensus.cc:2781] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:53.144569 11377 raft_consensus.cc:482] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:53.144589 11377 raft_consensus.cc:3037] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:53.145457 11377 raft_consensus.cc:504] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } }
I20260928 02:38:53.145604 11377 leader_election.cc:288] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [CANDIDATE]: Term 1 election: Requested vote from peers 57a6bfe4c06b4dd7a02cae25066042f0 (127.9.130.1:46599), 5389e19e864741a0ae58f47ad32297fa (127.9.130.2:34219)
I20260928 02:38:53.145735 11148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95d5581947c42db982f75baa92e4810" candidate_uuid: "e04e8766f8b648fd89b3b0463f888928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5389e19e864741a0ae58f47ad32297fa"
I20260928 02:38:53.145759 11008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95d5581947c42db982f75baa92e4810" candidate_uuid: "e04e8766f8b648fd89b3b0463f888928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a6bfe4c06b4dd7a02cae25066042f0"
I20260928 02:38:53.145803 11148 raft_consensus.cc:3037] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:53.145833 11008 raft_consensus.cc:3037] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:53.146590 11008 raft_consensus.cc:2454] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e04e8766f8b648fd89b3b0463f888928 in term 1.
I20260928 02:38:53.146593 11148 raft_consensus.cc:2454] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e04e8766f8b648fd89b3b0463f888928 in term 1.
I20260928 02:38:53.146771 11222 leader_election.cc:302] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57a6bfe4c06b4dd7a02cae25066042f0, e04e8766f8b648fd89b3b0463f888928; no voters:
I20260928 02:38:53.146842 11377 raft_consensus.cc:2781] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:53.146874 11377 raft_consensus.cc:686] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [term 1 LEADER]: Becoming Leader. State: Replica: e04e8766f8b648fd89b3b0463f888928, State: Running, Role: LEADER
I20260928 02:38:53.146919 11377 consensus_queue.cc:237] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } }
I20260928 02:38:53.147506 10850 catalog_manager.cc:5495] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 reported cstate change: term changed from 0 to 1, leader changed from <none> to e04e8766f8b648fd89b3b0463f888928 (127.9.130.3). New cstate: current_term: 1 leader_uuid: "e04e8766f8b648fd89b3b0463f888928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } health_report { overall_health: HEALTHY } } }
W20260928 02:38:53.153422 11055 tablet.cc:2369] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:38:53 dist-test-slave-kkrh krb5kdc[10806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:53.196022 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.188671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.9.130.62:40663)
0928 02:38:53.188739 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:53.188742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:53.188864 (+ 122us) server_negotiation.cc:407] Connection header received
0928 02:38:53.188944 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:53.188945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:53.188956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:53.189041 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:53.189491 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.190160 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.191217 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.191420 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.191510 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.193714 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:53.193715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:53.193716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:53.193722 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:53.194933 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:53.195269 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:53.195270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:53.195270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:53.195318 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:53.195549 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:53.195549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:53.195549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:53.195650 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:53.195705 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:53.195869 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 02:38:53.195952 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:38:53.196105 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.188617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:40663 (local address 127.0.0.1:54788)
0928 02:38:53.188785 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:53.188799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:53.188884 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:53.189049 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:53.189049 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:53.189364 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:53.189406 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.189408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.190257 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.190258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.191135 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.191138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.191504 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.191505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.191554 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.192110 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:53.192115 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:53.193632 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:53.195009 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:53.195014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:53.195017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:53.195205 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:53.195376 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:53.195377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:53.195377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:53.195452 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:53.195709 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:53.195714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:53.195758 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:38:53.195811 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 02:38:53.196022 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":70,"thread_start_us":71,"threads_started":1}
I20260928 02:38:53.198120 10850 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:54788:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:38:53.198351 10850 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:53.199760 11268 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf6d048b2d34d5cbaa2592159c8a4c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18469f2542494fc69bd7f20520629f05]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:38:53.199760 11128 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf6d048b2d34d5cbaa2592159c8a4c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18469f2542494fc69bd7f20520629f05]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:38:53.199797 10988 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf6d048b2d34d5cbaa2592159c8a4c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18469f2542494fc69bd7f20520629f05]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:38:53.199940 11268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf6d048b2d34d5cbaa2592159c8a4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:53.199940 11128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf6d048b2d34d5cbaa2592159c8a4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:53.200007 10988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf6d048b2d34d5cbaa2592159c8a4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:53.201327 11358 tablet_bootstrap.cc:492] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: Bootstrap starting.
I20260928 02:38:53.201335 11360 tablet_bootstrap.cc:492] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa: Bootstrap starting.
I20260928 02:38:53.201694 11358 tablet_bootstrap.cc:654] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:53.201715 11360 tablet_bootstrap.cc:654] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:53.201905 11357 tablet_bootstrap.cc:492] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0: Bootstrap starting.
I20260928 02:38:53.202304 11357 tablet_bootstrap.cc:654] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:53.202610 11358 tablet_bootstrap.cc:492] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: No bootstrap required, opened a new log
I20260928 02:38:53.202672 11358 ts_tablet_manager.cc:1397] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:53.202813 11358 raft_consensus.cc:348] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.202915 11360 tablet_bootstrap.cc:492] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa: No bootstrap required, opened a new log
I20260928 02:38:53.202925 11357 tablet_bootstrap.cc:492] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0: No bootstrap required, opened a new log
I20260928 02:38:53.202948 11358 raft_consensus.cc:374] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:53.202963 11357 ts_tablet_manager.cc:1397] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:53.202958 11360 ts_tablet_manager.cc:1397] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:53.202975 11358 raft_consensus.cc:729] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04e8766f8b648fd89b3b0463f888928, State: Initialized, Role: FOLLOWER
I20260928 02:38:53.203022 11358 consensus_queue.cc:260] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.203118 11358 ts_tablet_manager.cc:1428] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:53.203104 11360 raft_consensus.cc:348] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.203104 11357 raft_consensus.cc:348] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.203166 11360 raft_consensus.cc:374] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:53.203168 11357 raft_consensus.cc:374] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:53.203189 11357 raft_consensus.cc:729] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a6bfe4c06b4dd7a02cae25066042f0, State: Initialized, Role: FOLLOWER
I20260928 02:38:53.203189 11360 raft_consensus.cc:729] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5389e19e864741a0ae58f47ad32297fa, State: Initialized, Role: FOLLOWER
I20260928 02:38:53.203135 11367 tablet_replica.cc:439] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } }
I20260928 02:38:53.203217 11367 tablet_replica.cc:442] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:38:53.203230 11360 consensus_queue.cc:260] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.203234 11357 consensus_queue.cc:260] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.203313 11357 ts_tablet_manager.cc:1428] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:53.203313 11360 ts_tablet_manager.cc:1428] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:53.203429 11364 tablet_replica.cc:439] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } }
I20260928 02:38:53.203462 11363 tablet_replica.cc:439] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } }
I20260928 02:38:53.203510 11364 tablet_replica.cc:442] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:38:53.203522 11363 tablet_replica.cc:442] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 02:38:53.214614 11335 tablet.cc:2369] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:38:53.362500 11367 raft_consensus.cc:482] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:53.362612 11367 raft_consensus.cc:504] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.362802 11367 leader_election.cc:288] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5389e19e864741a0ae58f47ad32297fa (127.9.130.2:34219), 57a6bfe4c06b4dd7a02cae25066042f0 (127.9.130.1:46599)
I20260928 02:38:53.363020 11148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf6d048b2d34d5cbaa2592159c8a4c1" candidate_uuid: "e04e8766f8b648fd89b3b0463f888928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5389e19e864741a0ae58f47ad32297fa" is_pre_election: true
I20260928 02:38:53.363116 11148 raft_consensus.cc:2454] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e04e8766f8b648fd89b3b0463f888928 in term 0.
I20260928 02:38:53.363124 11008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf6d048b2d34d5cbaa2592159c8a4c1" candidate_uuid: "e04e8766f8b648fd89b3b0463f888928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" is_pre_election: true
I20260928 02:38:53.363262 11008 raft_consensus.cc:2454] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e04e8766f8b648fd89b3b0463f888928 in term 0.
I20260928 02:38:53.363291 11219 leader_election.cc:302] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5389e19e864741a0ae58f47ad32297fa, e04e8766f8b648fd89b3b0463f888928; no voters:
I20260928 02:38:53.363408 11367 raft_consensus.cc:2781] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:53.363462 11367 raft_consensus.cc:482] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:53.363492 11367 raft_consensus.cc:3037] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:53.364321 11367 raft_consensus.cc:504] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.364467 11367 leader_election.cc:288] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [CANDIDATE]: Term 1 election: Requested vote from peers 5389e19e864741a0ae58f47ad32297fa (127.9.130.2:34219), 57a6bfe4c06b4dd7a02cae25066042f0 (127.9.130.1:46599)
I20260928 02:38:53.364602 11148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf6d048b2d34d5cbaa2592159c8a4c1" candidate_uuid: "e04e8766f8b648fd89b3b0463f888928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5389e19e864741a0ae58f47ad32297fa"
I20260928 02:38:53.364691 11148 raft_consensus.cc:3037] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:53.364681 11008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf6d048b2d34d5cbaa2592159c8a4c1" candidate_uuid: "e04e8766f8b648fd89b3b0463f888928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a6bfe4c06b4dd7a02cae25066042f0"
I20260928 02:38:53.364771 11008 raft_consensus.cc:3037] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:53.365255 11148 raft_consensus.cc:2454] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e04e8766f8b648fd89b3b0463f888928 in term 1.
I20260928 02:38:53.365461 11219 leader_election.cc:302] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5389e19e864741a0ae58f47ad32297fa, e04e8766f8b648fd89b3b0463f888928; no voters:
I20260928 02:38:53.365477 11008 raft_consensus.cc:2454] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e04e8766f8b648fd89b3b0463f888928 in term 1.
I20260928 02:38:53.365559 11367 raft_consensus.cc:2781] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:53.365622 11367 raft_consensus.cc:686] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [term 1 LEADER]: Becoming Leader. State: Replica: e04e8766f8b648fd89b3b0463f888928, State: Running, Role: LEADER
I20260928 02:38:53.365675 11367 consensus_queue.cc:237] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } }
I20260928 02:38:53.365950 11377 tablet_replica.cc:439] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: TxnStatusTablet state changed. Reason: New leader e04e8766f8b648fd89b3b0463f888928. Latest consensus state: current_term: 1 leader_uuid: "e04e8766f8b648fd89b3b0463f888928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } }
I20260928 02:38:53.366032 11377 tablet_replica.cc:442] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:38:53.366210 11397 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:38:53.366318 11397 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:38:53.366266 10851 catalog_manager.cc:5495] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 reported cstate change: term changed from 0 to 1, leader changed from <none> to e04e8766f8b648fd89b3b0463f888928 (127.9.130.3). New cstate: current_term: 1 leader_uuid: "e04e8766f8b648fd89b3b0463f888928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } health_report { overall_health: UNKNOWN } } }
I20260928 02:38:53.385186 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.382620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.9.130.3:39563)
0928 02:38:53.382670 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:53.382671 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:53.382679 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:53.382758 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:53.382759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:53.382768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:53.382817 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:53.382961 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.383415 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.384533 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.384719 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.384783 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.384963 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:53.384989 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:53.385028 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:53.385110 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 02:38:53.385129 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:38:53.385234 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.382547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:39563 (local address 127.0.0.1:48416)
0928 02:38:53.382630 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:53.382641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:53.382684 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:53.382844 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:53.382845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:53.382846 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:38:53.382899 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.382901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.383486 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.383487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.384430 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.384432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.384797 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.384798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.384863 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.384880 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:38:53.385047 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:38:53.385054 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:38:53.385108 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:38:53.385152 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:38:53.386277 11008 raft_consensus.cc:1261] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 1 FOLLOWER]: Refusing update from remote peer e04e8766f8b648fd89b3b0463f888928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:53.386277 11148 raft_consensus.cc:1261] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa [term 1 FOLLOWER]: Refusing update from remote peer e04e8766f8b648fd89b3b0463f888928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:53.386512 11367 consensus_queue.cc:1035] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:53.386626 11377 consensus_queue.cc:1035] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:53.387152 11364 tablet_replica.cc:439] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa: TxnStatusTablet state changed. Reason: New leader e04e8766f8b648fd89b3b0463f888928. Latest consensus state: current_term: 1 leader_uuid: "e04e8766f8b648fd89b3b0463f888928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } }
I20260928 02:38:53.387245 11364 tablet_replica.cc:442] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:38:53.387233 11363 tablet_replica.cc:439] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0: TxnStatusTablet state changed. Reason: New leader e04e8766f8b648fd89b3b0463f888928. Latest consensus state: current_term: 1 leader_uuid: "e04e8766f8b648fd89b3b0463f888928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } }
I20260928 02:38:53.387336 11363 tablet_replica.cc:442] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:38:53.387919 11370 mvcc.cc:205] Tried to move back new op lower bound from 7334146594348195840 to 7334146594265866240. Current Snapshot: MvccSnapshot[applied={T|T < 7334146594348195840}]
I20260928 02:38:53.388000 11377 tablet_replica.cc:439] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e04e8766f8b648fd89b3b0463f888928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } }
I20260928 02:38:53.388079 11377 tablet_replica.cc:442] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:38:53.388121 11377 tablet_replica.cc:439] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e04e8766f8b648fd89b3b0463f888928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } }
I20260928 02:38:53.388157 11377 tablet_replica.cc:442] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P e04e8766f8b648fd89b3b0463f888928: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:38:53.388413 11364 tablet_replica.cc:439] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e04e8766f8b648fd89b3b0463f888928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } }
I20260928 02:38:53.388487 11364 tablet_replica.cc:442] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 5389e19e864741a0ae58f47ad32297fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:38:53.389070 11363 tablet_replica.cc:439] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e04e8766f8b648fd89b3b0463f888928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 } } peers { permanent_uuid: "e04e8766f8b648fd89b3b0463f888928" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39563 } } peers { permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 } } }
I20260928 02:38:53.389146 11363 tablet_replica.cc:442] T 4cf6d048b2d34d5cbaa2592159c8a4c1 P 57a6bfe4c06b4dd7a02cae25066042f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:38:53.394060 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.391900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:40295 (local address 127.9.130.3:39563)
0928 02:38:53.391931 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:53.391933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:53.391941 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:53.392026 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:53.392027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:53.392036 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:53.392088 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:53.392192 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.392653 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.393389 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.393579 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.393658 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.393815 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:53.393850 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:53.393910 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:53.393966 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:38:53.393990 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":17}
I20260928 02:38:53.394070 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.391810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:39563 (local address 127.9.130.3:40295)
0928 02:38:53.391892 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:53.391904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:53.391953 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:53.392093 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:53.392094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:53.392095 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:38:53.392136 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.392137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.392747 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.392748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.393307 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:53.393308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.393659 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.393660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:53.393708 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.393721 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:38:53.393909 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:38:53.393910 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:38:53.393963 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 02:38:53.394012 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":3}
I20260928 02:38:53.395937 11008 raft_consensus.cc:1261] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 1 FOLLOWER]: Refusing update from remote peer e04e8766f8b648fd89b3b0463f888928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:53.396072 11148 raft_consensus.cc:1261] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa [term 1 FOLLOWER]: Refusing update from remote peer e04e8766f8b648fd89b3b0463f888928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:53.396183 11365 consensus_queue.cc:1035] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57a6bfe4c06b4dd7a02cae25066042f0" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:53.396268 11367 consensus_queue.cc:1035] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5389e19e864741a0ae58f47ad32297fa" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:53.410035 10849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:53.410143 10849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:53.410727 10849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1d4d6e5e0ed497e99eb76f7e262743d: Sending DeleteTablet for 3 replicas of tablet e95d5581947c42db982f75baa92e4810
I20260928 02:38:53.411021 10988 tablet_service.cc:1514] Processing DeleteTablet for tablet e95d5581947c42db982f75baa92e4810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:53 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:59322
I20260928 02:38:53.411015 11128 tablet_service.cc:1514] Processing DeleteTablet for tablet e95d5581947c42db982f75baa92e4810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:53 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:37608
I20260928 02:38:53.411111 11268 tablet_service.cc:1514] Processing DeleteTablet for tablet e95d5581947c42db982f75baa92e4810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:53 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:48580
I20260928 02:38:53.411183 11382 tablet_replica.cc:330] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928: stopping tablet replica
I20260928 02:38:53.411229 11382 raft_consensus.cc:2229] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:38:53.411284 11382 raft_consensus.cc:2258] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:53.411512 11381 tablet_replica.cc:330] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0: stopping tablet replica
I20260928 02:38:53.411579 11381 raft_consensus.cc:2229] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:53.411624 11381 raft_consensus.cc:2258] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:53.411697 11380 tablet_replica.cc:330] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa: stopping tablet replica
I20260928 02:38:53.411755 11380 raft_consensus.cc:2229] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:53.411788 11380 raft_consensus.cc:2258] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:53.411967 11381 ts_tablet_manager.cc:1905] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:53.412016 11382 ts_tablet_manager.cc:1905] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:53.413614 11380 ts_tablet_manager.cc:1905] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:53.413792 11381 ts_tablet_manager.cc:1918] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:38:53.413846 11381 log.cc:1196] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-0/wal/wals/e95d5581947c42db982f75baa92e4810
I20260928 02:38:53.414059 11381 ts_tablet_manager.cc:1939] T e95d5581947c42db982f75baa92e4810 P 57a6bfe4c06b4dd7a02cae25066042f0: Deleting consensus metadata
I20260928 02:38:53.414994 10838 catalog_manager.cc:4841] TS 57a6bfe4c06b4dd7a02cae25066042f0 (127.9.130.1:46599): tablet e95d5581947c42db982f75baa92e4810 (table test-table [id=9ee818c50045465481e25425e83529f4]) successfully deleted
I20260928 02:38:53.415040 11382 ts_tablet_manager.cc:1918] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:38:53.415082 11382 log.cc:1196] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/e95d5581947c42db982f75baa92e4810
I20260928 02:38:53.415287 11382 ts_tablet_manager.cc:1939] T e95d5581947c42db982f75baa92e4810 P e04e8766f8b648fd89b3b0463f888928: Deleting consensus metadata
I20260928 02:38:53.415292 11380 ts_tablet_manager.cc:1918] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:38:53.415354 11380 log.cc:1196] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/e95d5581947c42db982f75baa92e4810
I20260928 02:38:53.415572 11380 ts_tablet_manager.cc:1939] T e95d5581947c42db982f75baa92e4810 P 5389e19e864741a0ae58f47ad32297fa: Deleting consensus metadata
I20260928 02:38:53.415611 10835 catalog_manager.cc:4841] TS e04e8766f8b648fd89b3b0463f888928 (127.9.130.3:39563): tablet e95d5581947c42db982f75baa92e4810 (table test-table [id=9ee818c50045465481e25425e83529f4]) successfully deleted
I20260928 02:38:53.415861 10835 catalog_manager.cc:4841] TS 5389e19e864741a0ae58f47ad32297fa (127.9.130.2:34219): tablet e95d5581947c42db982f75baa92e4810 (table test-table [id=9ee818c50045465481e25425e83529f4]) successfully deleted
Sep 28 02:38:53 dist-test-slave-kkrh krb5kdc[10806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563132, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.130.1@KRBTEST.COM
I20260928 02:38:53.421728 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.411649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.9.130.1:46599)
0928 02:38:53.411692 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:53.411694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:53.412049 (+ 355us) server_negotiation.cc:407] Connection header received
0928 02:38:53.412143 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:53.412144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:53.412154 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:53.412182 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:53.412928 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.413711 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.414800 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.415129 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.415171 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.418827 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:53.418834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:53.418834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:53.418841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:53.420573 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:53.420888 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:53.420891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:53.420892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:53.420976 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:53.421200 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:53.421201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:53.421201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:53.421356 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:53.421514 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:53.421564 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 02:38:53.421641 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 02:38:53.421974 11048 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51016
I20260928 02:38:53.429596 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:53.422838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37844 (local address 127.9.130.62:40663)
0928 02:38:53.422878 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:53.422880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:53.422922 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:38:53.422987 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:53.422988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:53.422998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:53.423060 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:53.423496 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.424270 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.425141 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:53.425405 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:53.425539 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:53.426766 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:53.426767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:53.426768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:53.426775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:53.428379 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:53.428773 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:53.428774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:53.428775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:53.428835 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:53.429058 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:53.429060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:53.429060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:53.429176 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:53.429291 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:53.429406 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 02:38:53.429488 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 02:38:53.429843 10849 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37844
I20260928 02:38:53.430575 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 10923
I20260928 02:38:53.439580 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 11063
I20260928 02:38:53.444660 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 11203
I20260928 02:38:53.450721 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 10819
2026-09-28T02:38:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2345 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:53 dist-test-slave-kkrh krb5kdc[11414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:53 dist-test-slave-kkrh krb5kdc[11414](info): set up 2 sockets
Sep 28 02:38:53 dist-test-slave-kkrh krb5kdc[11414](info): commencing operation
krb5kdc: starting...
W20260928 02:38:54.479663 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:38:54 dist-test-slave-kkrh krb5kdc[11414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:38:54.607594 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:46531
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:46531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:54.688916 11427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:54.689164 11427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:54.689213 11427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:54.690388 11427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:54.690460 11427 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:38:54.690482 11427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:54.690505 11427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:54.690532 11427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:54.690552 11427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:54.692070 11427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:46531
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:46531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:54.692348 11427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:54.692566 11427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:54.697443 11433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:54.697422 11435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:54.697605 11427 server_base.cc:1034] running on GCE node
W20260928 02:38:54.697422 11432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:54.697976 11427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:54.698212 11427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:54.699345 11427 hybrid_clock.cc:648] HybridClock initialized: now 1790563134699333 us; error 29 us; skew 500 ppm
Sep 28 02:38:54 dist-test-slave-kkrh krb5kdc[11414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563134, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:54.701730 11427 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:38:54.702178 11427 webserver.cc:460] Webserver started at http://127.9.130.62:44115/ using document root <none> and password file <none>
I20260928 02:38:54.702421 11427 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:54.702466 11427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:54.702628 11427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:54.703639 11427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "85b798dca7a54130b549933e8d9c4554"
format_stamp: "Formatted at 2026-09-28 02:38:54 on dist-test-slave-kkrh"
server_key: "b1ddb511bbd4d4228de690cf13359427"
server_key_iv: "0fb2818f74489fc05f3a1dc5618fae7b"
server_key_version: "encryptionkey@0"
I20260928 02:38:54.704015 11427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "85b798dca7a54130b549933e8d9c4554"
format_stamp: "Formatted at 2026-09-28 02:38:54 on dist-test-slave-kkrh"
server_key: "b1ddb511bbd4d4228de690cf13359427"
server_key_iv: "0fb2818f74489fc05f3a1dc5618fae7b"
server_key_version: "encryptionkey@0"
I20260928 02:38:54.705276 11427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:38:54.705979 11442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:54.706125 11427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:38:54.706214 11427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "85b798dca7a54130b549933e8d9c4554"
format_stamp: "Formatted at 2026-09-28 02:38:54 on dist-test-slave-kkrh"
server_key: "b1ddb511bbd4d4228de690cf13359427"
server_key_iv: "0fb2818f74489fc05f3a1dc5618fae7b"
server_key_version: "encryptionkey@0"
I20260928 02:38:54.706272 11427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:54.719135 11427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:54.721488 11427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:54.721596 11427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:54.725965 11427 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:46531
I20260928 02:38:54.726059 11504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:46531 every 8 connection(s)
I20260928 02:38:54.726372 11427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:54.727070 11505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:54.729323 11505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554: Bootstrap starting.
I20260928 02:38:54.729900 11505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:54.730175 11505 log.cc:824] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:54.730829 11505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554: No bootstrap required, opened a new log
I20260928 02:38:54.732455 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 11427
I20260928 02:38:54.732524 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:38:54.732697 9736 external_mini_cluster.cc:1428] Setting key 9bf79f3b91fefe08a7ccbae5391fbe0d
I20260928 02:38:54.733150 11505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b798dca7a54130b549933e8d9c4554" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 46531 } }
I20260928 02:38:54.733348 11505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:54.733399 11505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b798dca7a54130b549933e8d9c4554, State: Initialized, Role: FOLLOWER
I20260928 02:38:54.733529 11505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b798dca7a54130b549933e8d9c4554" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 46531 } }
I20260928 02:38:54.733603 11505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:54.733628 11505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:54.733668 11505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:54.734251 11505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b798dca7a54130b549933e8d9c4554" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 46531 } }
I20260928 02:38:54.734367 11505 leader_election.cc:302] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85b798dca7a54130b549933e8d9c4554; no voters:
I20260928 02:38:54.734545 11505 leader_election.cc:288] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:54.734565 11510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:54.734683 11510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [term 1 LEADER]: Becoming Leader. State: Replica: 85b798dca7a54130b549933e8d9c4554, State: Running, Role: LEADER
I20260928 02:38:54.734788 11505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:54.734766 11510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b798dca7a54130b549933e8d9c4554" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 46531 } }
I20260928 02:38:54.735105 11510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85b798dca7a54130b549933e8d9c4554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b798dca7a54130b549933e8d9c4554" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 46531 } } }
I20260928 02:38:54.735189 11510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:54.735321 11510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85b798dca7a54130b549933e8d9c4554. Latest consensus state: current_term: 1 leader_uuid: "85b798dca7a54130b549933e8d9c4554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b798dca7a54130b549933e8d9c4554" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 46531 } } }
I20260928 02:38:54.735387 11510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:54.735514 11516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:54.736234 11516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:54.737775 11516 catalog_manager.cc:1347] Generated new cluster ID: cce1ccc7530744128d14cd215c22be54
I20260928 02:38:54.737821 11516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:38:54 dist-test-slave-kkrh krb5kdc[11414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:54.745932 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:54.733233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.9.130.62:46531)
0928 02:38:54.733451 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:54.733460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:54.733491 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:38:54.733636 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:54.733643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:54.733698 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:54.733885 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:54.734501 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:54.735876 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:54.736828 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:54.737431 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:54.737566 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:54.742308 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:54.742312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:54.742313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:54.742324 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:54.744324 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:54.744712 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:54.744715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:54.744716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:54.744836 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:54.745034 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:54.745035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:54.745035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:54.745338 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:54.745500 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:54.745579 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:38:54.745725 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
I20260928 02:38:54.756184 11516 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:54.756884 11516 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:54.761803 11516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554: Generated new TSK 0
I20260928 02:38:54.761968 11516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:38:54.791157 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:46531
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:54.870798 11531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:54.870992 11531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:54.871016 11531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:54.872176 11531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:54.872216 11531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:54.872236 11531 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:38:54.872248 11531 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:38:54.872291 11531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:38:54.873741 11531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.130.62:46531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:54.874008 11531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:54.874234 11531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:54.879374 11536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:54.879374 11537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:54.879491 11539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:54.879503 11531 server_base.cc:1034] running on GCE node
I20260928 02:38:54.879892 11531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:54.880125 11531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:54.881282 11531 hybrid_clock.cc:648] HybridClock initialized: now 1790563134881270 us; error 34 us; skew 500 ppm
Sep 28 02:38:54 dist-test-slave-kkrh krb5kdc[11414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563134, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:54.883755 11531 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:38:54.884189 11531 webserver.cc:460] Webserver started at http://127.9.130.1:39589/ using document root <none> and password file <none>
I20260928 02:38:54.884430 11531 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:54.884500 11531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:54.884647 11531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:54.885629 11531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "c264570eeb9b42d8afa1ca82cc174096"
format_stamp: "Formatted at 2026-09-28 02:38:54 on dist-test-slave-kkrh"
server_key: "4206c8ad24b756ad03a2678c4f3f7887"
server_key_iv: "7c4dae15d2ed200fa68239bd76fdaa73"
server_key_version: "encryptionkey@0"
I20260928 02:38:54.885965 11531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "c264570eeb9b42d8afa1ca82cc174096"
format_stamp: "Formatted at 2026-09-28 02:38:54 on dist-test-slave-kkrh"
server_key: "4206c8ad24b756ad03a2678c4f3f7887"
server_key_iv: "7c4dae15d2ed200fa68239bd76fdaa73"
server_key_version: "encryptionkey@0"
I20260928 02:38:54.887199 11531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:54.887938 11546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:54.888106 11531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:54.888202 11531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "c264570eeb9b42d8afa1ca82cc174096"
format_stamp: "Formatted at 2026-09-28 02:38:54 on dist-test-slave-kkrh"
server_key: "4206c8ad24b756ad03a2678c4f3f7887"
server_key_iv: "7c4dae15d2ed200fa68239bd76fdaa73"
server_key_version: "encryptionkey@0"
I20260928 02:38:54.888298 11531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:54.899363 11531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:54.901122 11531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:54.901281 11531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:54.901744 11531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:54.901805 11531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:54.901849 11531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:54.901873 11531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:38:54 dist-test-slave-kkrh krb5kdc[11414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563134, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:54.908811 11531 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:43053
I20260928 02:38:54.908928 11661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:43053 every 8 connection(s)
I20260928 02:38:54.909192 11531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:38:54.911883 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:54.902257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:41085 (local address 127.9.130.62:46531)
0928 02:38:54.902327 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:54.902330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:54.902470 (+ 140us) server_negotiation.cc:407] Connection header received
0928 02:38:54.902956 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:54.902958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:54.902975 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:54.903055 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:54.903909 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:54.904744 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:54.905820 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:54.906139 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:54.906222 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:54.909070 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:54.909072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:54.909072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:54.909079 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:54.910503 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:54.910903 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:54.910904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:54.910905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:54.910964 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:54.911215 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:54.911216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:54.911217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:54.911339 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:54.911411 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:54.911683 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 02:38:54.911767 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:38:54.912066 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:54.902213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:46531 (local address 127.9.130.1:41085)
0928 02:38:54.902362 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:54.902401 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:54.902864 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:54.903140 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:54.903144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:54.903471 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:54.903738 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:54.903745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:54.904861 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:54.904862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:54.905725 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:54.905728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:54.906271 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:54.906272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:54.906445 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:54.906984 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:54.906994 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:54.908985 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:54.910586 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:54.910592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:54.910594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:54.910822 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:54.911038 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:54.911039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:54.911040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:54.911146 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:54.911414 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:54.911420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:54.911550 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 02:38:54.911661 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:38:54.911844 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260928 02:38:54.912562 11662 heartbeater.cc:347] Connected to a master server at 127.9.130.62:46531
I20260928 02:38:54.912640 11662 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:54.912827 11662 heartbeater.cc:511] Master 127.9.130.62:46531 requested a full tablet report, sending...
I20260928 02:38:54.913368 11458 ts_manager.cc:194] Registered new tserver with Master: c264570eeb9b42d8afa1ca82cc174096 (127.9.130.1:43053)
I20260928 02:38:54.914119 11458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:41085
I20260928 02:38:54.915640 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 11531
I20260928 02:38:54.915757 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:38:54.915984 9736 external_mini_cluster.cc:1428] Setting key 682ce2870e9d7c8729884da6651552ad
I20260928 02:38:54.919654 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:54.913527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:46531 (local address 127.9.130.1:42551)
0928 02:38:54.913658 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:54.913669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:54.913719 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:54.914385 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:54.914386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:54.914579 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:54.914646 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:54.914648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:54.915256 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:54.915257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:54.915748 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:54.915758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:54.916281 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:54.916282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:54.916359 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:54.916794 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:54.916799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:54.917184 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:54.918698 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:54.918700 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:54.918700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:54.918924 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:54.919126 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:54.919127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:54.919127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:54.919176 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:54.919434 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:54.919435 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:54.919468 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:38:54.919518 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:38:54.919587 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260928 02:38:54.919760 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:54.913610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:42551 (local address 127.9.130.62:46531)
0928 02:38:54.914273 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:54.914275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:54.914282 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:38:54.914296 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:54.914296 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:54.914303 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:54.914378 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:54.914720 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:54.915168 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:54.915894 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:54.916170 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:54.916279 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:54.917273 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:54.917274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:54.917274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:54.917280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:54.918617 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:54.919003 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:54.919004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:54.919004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:54.919061 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:54.919236 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:54.919237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:54.919238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:54.919373 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:54.919440 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:54.919541 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 02:38:54.919621 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649}
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:38:54.942657 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:46531
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:55.022516 11671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:55.022758 11671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:55.022797 11671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:55.023950 11671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:55.024015 11671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:55.024049 11671 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:38:55.024078 11671 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:38:55.024114 11671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:38:55.025556 11671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.130.62:46531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:55.025818 11671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:55.026050 11671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:55.031006 11676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:55.031042 11677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:55.031179 11671 server_base.cc:1034] running on GCE node
W20260928 02:38:55.031072 11679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:55.031463 11671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:55.031694 11671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:55.032862 11671 hybrid_clock.cc:648] HybridClock initialized: now 1790563135032838 us; error 49 us; skew 500 ppm
Sep 28 02:38:55 dist-test-slave-kkrh krb5kdc[11414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563135, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:55.035363 11671 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:38:55.035959 11671 webserver.cc:460] Webserver started at http://127.9.130.2:43885/ using document root <none> and password file <none>
I20260928 02:38:55.036187 11671 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:55.036260 11671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:55.036397 11671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:55.037510 11671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "462a4a4221774bdfb2d2a129816b7fc7"
format_stamp: "Formatted at 2026-09-28 02:38:55 on dist-test-slave-kkrh"
server_key: "18b7ce757dc4808eb669c0a4c42029a9"
server_key_iv: "5c92d61eefe7d9f29def6d9af8514e59"
server_key_version: "encryptionkey@0"
I20260928 02:38:55.037879 11671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "462a4a4221774bdfb2d2a129816b7fc7"
format_stamp: "Formatted at 2026-09-28 02:38:55 on dist-test-slave-kkrh"
server_key: "18b7ce757dc4808eb669c0a4c42029a9"
server_key_iv: "5c92d61eefe7d9f29def6d9af8514e59"
server_key_version: "encryptionkey@0"
I20260928 02:38:55.039160 11671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:55.039901 11686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:55.040066 11671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:55.040166 11671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "462a4a4221774bdfb2d2a129816b7fc7"
format_stamp: "Formatted at 2026-09-28 02:38:55 on dist-test-slave-kkrh"
server_key: "18b7ce757dc4808eb669c0a4c42029a9"
server_key_iv: "5c92d61eefe7d9f29def6d9af8514e59"
server_key_version: "encryptionkey@0"
I20260928 02:38:55.040261 11671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:55.057255 11671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:55.059682 11671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:55.059872 11671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:55.060389 11671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:55.060451 11671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:55.060496 11671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:55.060525 11671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:38:55 dist-test-slave-kkrh krb5kdc[11414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563135, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:55.067513 11671 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:35417
I20260928 02:38:55.067898 11671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:38:55.067996 11801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:35417 every 8 connection(s)
I20260928 02:38:55.070729 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.060897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:49807 (local address 127.9.130.62:46531)
0928 02:38:55.060939 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.060941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.061150 (+ 209us) server_negotiation.cc:407] Connection header received
0928 02:38:55.061669 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.061670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.061682 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.061738 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:55.062615 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.063439 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.064764 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.065199 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.065275 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.067960 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:55.067962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:55.067963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:55.067970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:55.069308 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:55.069781 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:55.069782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:55.069782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:55.069835 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:55.070104 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:55.070105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:55.070105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:55.070210 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:55.070267 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.070541 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.070616 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:38:55.070751 11695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.060881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:46531 (local address 127.9.130.2:49807)
0928 02:38:55.061032 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.061066 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.061566 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.061827 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.061830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.062189 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:55.062500 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.062506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.063537 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.063538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.064632 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.064635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.065347 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.065348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.065514 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.066002 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:55.066014 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:55.067861 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:55.069402 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:55.069408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:55.069410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:55.069669 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:55.069898 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:55.069898 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:55.069899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:55.070010 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:55.070270 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:55.070276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:55.070391 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.070476 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.070594 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":92,"threads_started":1}
I20260928 02:38:55.071388 11802 heartbeater.cc:347] Connected to a master server at 127.9.130.62:46531
I20260928 02:38:55.071470 11802 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:55.071646 11802 heartbeater.cc:511] Master 127.9.130.62:46531 requested a full tablet report, sending...
I20260928 02:38:55.072085 11459 ts_manager.cc:194] Registered new tserver with Master: 462a4a4221774bdfb2d2a129816b7fc7 (127.9.130.2:35417)
I20260928 02:38:55.072615 11459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:49807
I20260928 02:38:55.077286 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 11671
I20260928 02:38:55.077386 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:38:55.077598 9736 external_mini_cluster.cc:1428] Setting key 329de45f57eeaaa49c43ea8eee0a0383
I20260928 02:38:55.079694 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.072549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:34995 (local address 127.9.130.62:46531)
0928 02:38:55.072597 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.072599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.072660 (+ 61us) server_negotiation.cc:407] Connection header received
0928 02:38:55.072739 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.072740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.072750 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.072812 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:55.073167 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.073848 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.074537 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.074871 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.074943 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.076427 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:55.076428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:55.076428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:55.076436 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:55.078299 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:55.078735 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:55.078737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:55.078737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:55.078805 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:55.079041 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:55.079042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:55.079042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:55.079229 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:55.079287 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.079513 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.079590 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:38:55.080276 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.072449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:46531 (local address 127.9.130.2:34995)
0928 02:38:55.072586 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.072596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.072664 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.072818 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.072819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.073029 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:55.073089 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.073091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.073948 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.073949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.074439 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.074441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.074996 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.074997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.075074 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.075903 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:55.075908 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:55.076321 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:55.078387 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:55.078388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:55.078389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:55.078616 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:55.078894 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:55.078896 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:55.078896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:55.078966 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:55.079339 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:55.079340 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:55.079392 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.080024 (+ 632us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.080143 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":5,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:38:55.102684 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:46531
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:55.183351 11811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:55.183552 11811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:55.183576 11811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:55.184827 11811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:55.184878 11811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:55.184900 11811 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:38:55.184911 11811 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:38:55.184928 11811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:38:55.186393 11811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.130.62:46531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:55.186633 11811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:55.186854 11811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:55.191681 11816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:55.191673 11817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:55.191685 11819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:55.191918 11811 server_base.cc:1034] running on GCE node
I20260928 02:38:55.192119 11811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:55.192369 11811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:55.193508 11811 hybrid_clock.cc:648] HybridClock initialized: now 1790563135193502 us; error 49 us; skew 500 ppm
Sep 28 02:38:55 dist-test-slave-kkrh krb5kdc[11414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563135, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:55.195981 11811 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:38:55.196411 11811 webserver.cc:460] Webserver started at http://127.9.130.3:43115/ using document root <none> and password file <none>
I20260928 02:38:55.196591 11811 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:55.196640 11811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:55.196732 11811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:55.197768 11811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "13f45b3e75f0441b8b351500acfab35f"
format_stamp: "Formatted at 2026-09-28 02:38:55 on dist-test-slave-kkrh"
server_key: "f3ffa0f1d031340be066521e117df616"
server_key_iv: "d6a3f43bd6b1eef78caaab1b48164e6e"
server_key_version: "encryptionkey@0"
I20260928 02:38:55.198125 11811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "13f45b3e75f0441b8b351500acfab35f"
format_stamp: "Formatted at 2026-09-28 02:38:55 on dist-test-slave-kkrh"
server_key: "f3ffa0f1d031340be066521e117df616"
server_key_iv: "d6a3f43bd6b1eef78caaab1b48164e6e"
server_key_version: "encryptionkey@0"
I20260928 02:38:55.199496 11811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:55.200460 11826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:55.200685 11811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:55.200798 11811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "13f45b3e75f0441b8b351500acfab35f"
format_stamp: "Formatted at 2026-09-28 02:38:55 on dist-test-slave-kkrh"
server_key: "f3ffa0f1d031340be066521e117df616"
server_key_iv: "d6a3f43bd6b1eef78caaab1b48164e6e"
server_key_version: "encryptionkey@0"
I20260928 02:38:55.200892 11811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:55.217362 11811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:55.219484 11811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:55.219719 11811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:55.220422 11811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:55.220492 11811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:55.220526 11811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:55.220556 11811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:38:55 dist-test-slave-kkrh krb5kdc[11414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563135, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:55.228014 11811 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:39471
I20260928 02:38:55.228376 11811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:38:55.228924 11941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:39471 every 8 connection(s)
I20260928 02:38:55.230113 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.220844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:51793 (local address 127.9.130.62:46531)
0928 02:38:55.220895 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.220897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.221142 (+ 245us) server_negotiation.cc:407] Connection header received
0928 02:38:55.221614 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.221616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.221630 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.221729 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:55.222468 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.222990 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.224079 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.224288 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.224371 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.226930 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:55.226932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:55.226932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:55.226939 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:55.228581 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:55.229063 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:55.229064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:55.229065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:55.229124 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:55.229371 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:55.229372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:55.229373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:55.229488 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:55.229564 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.229886 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.230014 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:38:55.230343 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.220841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:46531 (local address 127.9.130.3:51793)
0928 02:38:55.221003 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.221042 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.221513 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.221817 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.221820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.222060 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:55.222346 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.222352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.223083 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.223084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.223979 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.223982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.224381 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.224382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.224504 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.225014 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:55.225022 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:55.226807 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:55.228672 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:55.228679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:55.228682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:55.228988 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:55.229175 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:55.229176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:55.229176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:55.229292 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:55.229555 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:55.229560 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:55.229698 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.229933 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.230106 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260928 02:38:55.230854 11942 heartbeater.cc:347] Connected to a master server at 127.9.130.62:46531
I20260928 02:38:55.230957 11942 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:55.231123 11942 heartbeater.cc:511] Master 127.9.130.62:46531 requested a full tablet report, sending...
I20260928 02:38:55.231468 11458 ts_manager.cc:194] Registered new tserver with Master: 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471)
I20260928 02:38:55.231961 11458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:51793
I20260928 02:38:55.237123 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.231762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:46531 (local address 127.9.130.3:54415)
0928 02:38:55.231956 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.231967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.232029 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.232156 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.232157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.232373 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:55.232436 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.232438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.233088 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.233089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.233569 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.233570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.233890 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.233891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.233947 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.234310 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:55.234314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:55.234564 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:55.236370 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:55.236371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:55.236371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:55.236539 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:55.236680 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:55.236680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:55.236680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:55.236722 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:55.236907 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:55.236907 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:55.236939 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.236985 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.237055 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":112,"threads_started":1}
I20260928 02:38:55.237133 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.231865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:54415 (local address 127.9.130.62:46531)
0928 02:38:55.231898 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.231900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.232037 (+ 137us) server_negotiation.cc:407] Connection header received
0928 02:38:55.232085 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.232086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.232096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.232146 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:55.232530 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.233000 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.233622 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.233826 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.233883 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.234633 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:55.234634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:55.234634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:55.234639 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:55.236295 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:55.236598 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:55.236599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:55.236599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:55.236643 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:55.236761 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:55.236761 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:55.236761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:55.236841 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:55.236911 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.237014 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.237079 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 02:38:55.237820 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 11811
I20260928 02:38:55.237900 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:38:55.238116 9736 external_mini_cluster.cc:1428] Setting key d9d58adbfa1b1e21ca4c78343b57dc3c
I20260928 02:38:55.238989 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:38:55 dist-test-slave-kkrh krb5kdc[11414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:38:55 dist-test-slave-kkrh krb5kdc[11414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:55.260958 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.253920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.9.130.62:46531)
0928 02:38:55.253984 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.253986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.253992 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:38:55.254015 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.254016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.254023 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.254089 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:55.254631 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.255114 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.255899 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.256084 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.256188 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.258340 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:55.258342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:55.258342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:55.258348 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:55.259797 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:55.260206 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:55.260207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:55.260208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:55.260264 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:55.260466 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:55.260467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:55.260467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:55.260571 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:55.260674 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.260784 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.260858 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:38:55.262454 11458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:38:55.264022 11458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:55.270721 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.266589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.9.130.1:43053)
0928 02:38:55.266795 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.266800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.266836 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:38:55.266988 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.266992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.267009 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.267077 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:55.267307 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.268606 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.269746 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.270293 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.270373 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.270434 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.270584 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.270621 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
I20260928 02:38:55.270764 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.266387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:43053 (local address 127.0.0.1:33940)
0928 02:38:55.266692 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.266733 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.266857 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.267146 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.267149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.267151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:55.267224 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.267226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.268699 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.268700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.269658 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.269659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.270383 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.270384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.270463 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.270479 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.270566 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.270618 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":45,"threads_started":1}
I20260928 02:38:55.270987 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.266552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.9.130.3:39471)
0928 02:38:55.266786 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.266793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.266984 (+ 191us) server_negotiation.cc:407] Connection header received
0928 02:38:55.267029 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.267032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.267051 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.267146 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:55.267441 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.268171 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.269478 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.270328 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.270402 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.270456 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.270831 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.270877 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260928 02:38:55.270789 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.266463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:39471 (local address 127.0.0.1:44882)
0928 02:38:55.266846 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.266857 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.266922 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.267147 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.267314 (+ 167us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.267316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:55.267357 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.267359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.268255 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.268256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.269391 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.269394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.270492 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.270493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.270559 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.270563 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.270616 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.270658 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":94,"threads_started":1}
I20260928 02:38:55.271164 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.266560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:35417 (local address 127.0.0.1:59208)
0928 02:38:55.266948 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.266958 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.267009 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.267278 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.267279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.267281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:55.267338 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.267341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.268739 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.268740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.269881 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.269884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.270916 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.270917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.270991 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.270995 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.271052 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.271088 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":75,"threads_started":1}
I20260928 02:38:55.271435 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.266797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.9.130.2:35417)
0928 02:38:55.266942 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.266947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.267004 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:38:55.267078 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.267080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.267095 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.267186 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:55.267435 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.268646 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.269982 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.270760 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.270827 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.270882 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.271231 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.271327 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260928 02:38:55.271994 11876 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbd79220b664f9f8f4aa1f3b4760d8c (DEFAULT_TABLE table=test-table [id=9fc65c69f3ea4718a276da477f962958]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:55.272073 11736 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbd79220b664f9f8f4aa1f3b4760d8c (DEFAULT_TABLE table=test-table [id=9fc65c69f3ea4718a276da477f962958]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:55.272079 11596 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbd79220b664f9f8f4aa1f3b4760d8c (DEFAULT_TABLE table=test-table [id=9fc65c69f3ea4718a276da477f962958]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:55.272393 11736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbd79220b664f9f8f4aa1f3b4760d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:55.272413 11596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbd79220b664f9f8f4aa1f3b4760d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:55.272416 11876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbd79220b664f9f8f4aa1f3b4760d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:55.275041 11965 tablet_bootstrap.cc:492] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096: Bootstrap starting.
I20260928 02:38:55.275629 11965 tablet_bootstrap.cc:654] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:55.275883 11966 tablet_bootstrap.cc:492] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f: Bootstrap starting.
I20260928 02:38:55.276129 11965 log.cc:824] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:55.276228 11967 tablet_bootstrap.cc:492] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7: Bootstrap starting.
I20260928 02:38:55.276912 11966 tablet_bootstrap.cc:654] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:55.276942 11967 tablet_bootstrap.cc:654] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:55.277019 11965 tablet_bootstrap.cc:492] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096: No bootstrap required, opened a new log
I20260928 02:38:55.277103 11965 ts_tablet_manager.cc:1397] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:55.277244 11966 log.cc:824] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:55.277302 11967 log.cc:824] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:55.277980 11967 tablet_bootstrap.cc:492] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7: No bootstrap required, opened a new log
I20260928 02:38:55.278041 11966 tablet_bootstrap.cc:492] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f: No bootstrap required, opened a new log
I20260928 02:38:55.278064 11967 ts_tablet_manager.cc:1397] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:55.278120 11966 ts_tablet_manager.cc:1397] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:55.279412 11965 raft_consensus.cc:348] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.279601 11965 raft_consensus.cc:374] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:55.279640 11965 raft_consensus.cc:729] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c264570eeb9b42d8afa1ca82cc174096, State: Initialized, Role: FOLLOWER
I20260928 02:38:55.279619 11967 raft_consensus.cc:348] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.279810 11967 raft_consensus.cc:374] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:55.279800 11965 consensus_queue.cc:260] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.279892 11967 raft_consensus.cc:729] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 462a4a4221774bdfb2d2a129816b7fc7, State: Initialized, Role: FOLLOWER
I20260928 02:38:55.280009 11967 consensus_queue.cc:260] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.280119 11965 ts_tablet_manager.cc:1428] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 02:38:55.280269 11967 ts_tablet_manager.cc:1428] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:38:55.280257 11662 heartbeater.cc:503] Master 127.9.130.62:46531 was elected leader, sending a full tablet report...
I20260928 02:38:55.280380 11802 heartbeater.cc:503] Master 127.9.130.62:46531 was elected leader, sending a full tablet report...
I20260928 02:38:55.280349 11966 raft_consensus.cc:348] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.280500 11966 raft_consensus.cc:374] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:55.280542 11966 raft_consensus.cc:729] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f45b3e75f0441b8b351500acfab35f, State: Initialized, Role: FOLLOWER
I20260928 02:38:55.280673 11966 consensus_queue.cc:260] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.280956 11966 ts_tablet_manager.cc:1428] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:38:55.280967 11942 heartbeater.cc:503] Master 127.9.130.62:46531 was elected leader, sending a full tablet report...
W20260928 02:38:55.318596 11803 tablet.cc:2369] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:38:55.410107 11663 tablet.cc:2369] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:38:55.461845 11972 raft_consensus.cc:482] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:55.461974 11972 raft_consensus.cc:504] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.462395 11972 leader_election.cc:288] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c264570eeb9b42d8afa1ca82cc174096 (127.9.130.1:43053), 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471)
I20260928 02:38:55.465477 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.462715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:34567 (local address 127.9.130.1:43053)
0928 02:38:55.462769 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.462774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.462938 (+ 164us) server_negotiation.cc:407] Connection header received
0928 02:38:55.463013 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.463014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.463026 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.463091 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:55.463314 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.463749 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.464754 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.465116 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.465168 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.465176 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.465363 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.465411 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:38:55.465468 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.462631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:43053 (local address 127.9.130.2:34567)
0928 02:38:55.462852 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.462862 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.462930 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.463144 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.463145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.463148 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:55.463226 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.463228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.463856 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.463857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.464658 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.464660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.465192 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.465193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.465273 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.465277 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.465328 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.465356 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":67,"thread_start_us":91,"threads_started":1}
I20260928 02:38:55.465539 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.462703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:44545 (local address 127.9.130.3:39471)
0928 02:38:55.462810 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.462812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.462822 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:38:55.462851 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.462852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.462865 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.462933 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:55.463171 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.463718 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.464810 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.465140 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.465230 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.465239 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.465427 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.465467 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 02:38:55.465556 11695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.462620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:39471 (local address 127.9.130.2:44545)
0928 02:38:55.462677 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.462690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.462751 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.462998 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.462999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.463001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:55.463077 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.463079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.463854 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.463856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.464660 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.464662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.465247 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.465248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.465343 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.465348 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.465404 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.465440 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":36}
I20260928 02:38:55.465852 11616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbd79220b664f9f8f4aa1f3b4760d8c" candidate_uuid: "462a4a4221774bdfb2d2a129816b7fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c264570eeb9b42d8afa1ca82cc174096" is_pre_election: true
I20260928 02:38:55.465855 11896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbd79220b664f9f8f4aa1f3b4760d8c" candidate_uuid: "462a4a4221774bdfb2d2a129816b7fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f45b3e75f0441b8b351500acfab35f" is_pre_election: true
I20260928 02:38:55.466004 11896 raft_consensus.cc:2454] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 462a4a4221774bdfb2d2a129816b7fc7 in term 0.
I20260928 02:38:55.466004 11616 raft_consensus.cc:2454] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 462a4a4221774bdfb2d2a129816b7fc7 in term 0.
I20260928 02:38:55.466236 11687 leader_election.cc:302] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 462a4a4221774bdfb2d2a129816b7fc7, c264570eeb9b42d8afa1ca82cc174096; no voters:
I20260928 02:38:55.466418 11972 raft_consensus.cc:2781] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:55.466498 11972 raft_consensus.cc:482] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:55.466532 11972 raft_consensus.cc:3037] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:55.467469 11972 raft_consensus.cc:504] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.467617 11972 leader_election.cc:288] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [CANDIDATE]: Term 1 election: Requested vote from peers c264570eeb9b42d8afa1ca82cc174096 (127.9.130.1:43053), 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471)
I20260928 02:38:55.467793 11896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbd79220b664f9f8f4aa1f3b4760d8c" candidate_uuid: "462a4a4221774bdfb2d2a129816b7fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f45b3e75f0441b8b351500acfab35f"
I20260928 02:38:55.467890 11616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbd79220b664f9f8f4aa1f3b4760d8c" candidate_uuid: "462a4a4221774bdfb2d2a129816b7fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c264570eeb9b42d8afa1ca82cc174096"
I20260928 02:38:55.467923 11896 raft_consensus.cc:3037] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:55.467984 11616 raft_consensus.cc:3037] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:55.468598 11896 raft_consensus.cc:2454] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 462a4a4221774bdfb2d2a129816b7fc7 in term 1.
I20260928 02:38:55.468680 11616 raft_consensus.cc:2454] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 462a4a4221774bdfb2d2a129816b7fc7 in term 1.
I20260928 02:38:55.468783 11689 leader_election.cc:302] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f45b3e75f0441b8b351500acfab35f, 462a4a4221774bdfb2d2a129816b7fc7; no voters:
I20260928 02:38:55.468869 11972 raft_consensus.cc:2781] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:55.468997 11972 raft_consensus.cc:686] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 462a4a4221774bdfb2d2a129816b7fc7, State: Running, Role: LEADER
I20260928 02:38:55.469086 11972 consensus_queue.cc:237] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.469913 11458 catalog_manager.cc:5495] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 462a4a4221774bdfb2d2a129816b7fc7 (127.9.130.2). New cstate: current_term: 1 leader_uuid: "462a4a4221774bdfb2d2a129816b7fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } health_report { overall_health: UNKNOWN } } }
W20260928 02:38:55.479090 11943 tablet.cc:2369] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:38:55 dist-test-slave-kkrh krb5kdc[11414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563134, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:55.535375 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.528608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.9.130.62:46531)
0928 02:38:55.528659 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.528661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.528781 (+ 120us) server_negotiation.cc:407] Connection header received
0928 02:38:55.528841 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.528842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.528851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.528932 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:55.529393 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.529882 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.530611 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.530844 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.530925 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.533037 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:55.533038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:55.533038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:55.533045 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:55.534185 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:55.534554 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:55.534556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:55.534556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:55.534611 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:55.534842 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:55.534843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:55.534844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:55.534951 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:55.535025 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.535172 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.535255 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:38:55.535426 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.528547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:46531 (local address 127.0.0.1:53098)
0928 02:38:55.528690 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.528701 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.528786 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.528938 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.528939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.529262 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:55.529306 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.529308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.529963 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.529964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.530525 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.530527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.530930 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.530931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.530989 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.531452 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:55.531456 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:55.532949 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:55.534280 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:55.534287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:55.534289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:55.534469 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:55.534713 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:55.534714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:55.534714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:55.534773 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:55.535015 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:55.535021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:55.535064 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.535152 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.535308 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":55,"thread_start_us":63,"threads_started":1}
I20260928 02:38:55.537510 11458 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:53098:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:38:55.537714 11458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:55.539168 11736 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1d49df507449e4bacb078234f77fb4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc9f5220ebb043a2a5c023988fa052b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:38:55.539181 11876 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1d49df507449e4bacb078234f77fb4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc9f5220ebb043a2a5c023988fa052b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:38:55.539187 11596 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1d49df507449e4bacb078234f77fb4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc9f5220ebb043a2a5c023988fa052b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:38:55.539335 11736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1d49df507449e4bacb078234f77fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:55.539343 11876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1d49df507449e4bacb078234f77fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:55.539347 11596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1d49df507449e4bacb078234f77fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:55.540719 11965 tablet_bootstrap.cc:492] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096: Bootstrap starting.
I20260928 02:38:55.540817 11967 tablet_bootstrap.cc:492] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7: Bootstrap starting.
I20260928 02:38:55.540823 11966 tablet_bootstrap.cc:492] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f: Bootstrap starting.
I20260928 02:38:55.541134 11965 tablet_bootstrap.cc:654] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:55.541198 11967 tablet_bootstrap.cc:654] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:55.541245 11966 tablet_bootstrap.cc:654] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:55.541775 11965 tablet_bootstrap.cc:492] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096: No bootstrap required, opened a new log
I20260928 02:38:55.541831 11965 ts_tablet_manager.cc:1397] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:38:55.541903 11967 tablet_bootstrap.cc:492] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7: No bootstrap required, opened a new log
I20260928 02:38:55.541953 11967 ts_tablet_manager.cc:1397] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:38:55.542008 11965 raft_consensus.cc:348] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.542079 11965 raft_consensus.cc:374] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:55.542094 11965 raft_consensus.cc:729] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c264570eeb9b42d8afa1ca82cc174096, State: Initialized, Role: FOLLOWER
I20260928 02:38:55.542119 11967 raft_consensus.cc:348] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.542146 11965 consensus_queue.cc:260] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.542191 11967 raft_consensus.cc:374] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:55.542223 11967 raft_consensus.cc:729] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 462a4a4221774bdfb2d2a129816b7fc7, State: Initialized, Role: FOLLOWER
I20260928 02:38:55.542253 11965 ts_tablet_manager.cc:1428] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 02:38:55.542272 11967 consensus_queue.cc:260] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.542393 11967 ts_tablet_manager.cc:1428] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 02:38:55.542382 11971 tablet_replica.cc:439] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } } }
I20260928 02:38:55.542456 11971 tablet_replica.cc:442] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:38:55.542476 11972 tablet_replica.cc:439] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } } }
I20260928 02:38:55.542557 11972 tablet_replica.cc:442] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:38:55.543058 11966 tablet_bootstrap.cc:492] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f: No bootstrap required, opened a new log
I20260928 02:38:55.543107 11966 ts_tablet_manager.cc:1397] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:55.543247 11966 raft_consensus.cc:348] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.543320 11966 raft_consensus.cc:374] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:55.543350 11966 raft_consensus.cc:729] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f45b3e75f0441b8b351500acfab35f, State: Initialized, Role: FOLLOWER
I20260928 02:38:55.543392 11966 consensus_queue.cc:260] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.543493 11966 ts_tablet_manager.cc:1428] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:55.543567 11973 tablet_replica.cc:439] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } } }
I20260928 02:38:55.543642 11973 tablet_replica.cc:442] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:38:55.882344 11971 raft_consensus.cc:482] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:55.882447 11971 raft_consensus.cc:504] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.882900 11971 leader_election.cc:288] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 462a4a4221774bdfb2d2a129816b7fc7 (127.9.130.2:35417), 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471)
I20260928 02:38:55.887025 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.883199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:47643 (local address 127.9.130.3:39471)
0928 02:38:55.883253 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.883256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.883350 (+ 94us) server_negotiation.cc:407] Connection header received
0928 02:38:55.883382 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.883383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.883394 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.883464 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:55.883678 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.884464 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.886037 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.886596 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.886676 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.886686 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.886902 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.886960 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:38:55.887027 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.883113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:39471 (local address 127.9.130.1:47643)
0928 02:38:55.883259 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.883268 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.883324 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.883501 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.883502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.883506 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:55.883600 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.883601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.884605 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.884607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.885927 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.885930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.886702 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.886703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.886816 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.886820 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.886880 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.886929 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":69,"thread_start_us":64,"threads_started":1}
I20260928 02:38:55.887048 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.883073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:35417 (local address 127.9.130.1:59133)
0928 02:38:55.883217 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.883228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.883293 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.883501 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.883502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.883505 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:55.883594 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.883596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.884607 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.884607 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.886067 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.886069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.886821 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.886822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.886899 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.886901 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.886950 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.886980 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":18,"thread_start_us":62,"threads_started":1}
I20260928 02:38:55.887311 11988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.883155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:59133 (local address 127.9.130.2:35417)
0928 02:38:55.883261 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.883263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.883275 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:38:55.883395 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.883396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.883407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.883479 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:55.883677 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.884464 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.886160 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.886743 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.886800 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.886808 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.886969 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.887007 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":43,"threads_started":1}
I20260928 02:38:55.887387 11896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1d49df507449e4bacb078234f77fb4" candidate_uuid: "c264570eeb9b42d8afa1ca82cc174096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f45b3e75f0441b8b351500acfab35f" is_pre_election: true
I20260928 02:38:55.887498 11896 raft_consensus.cc:2454] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c264570eeb9b42d8afa1ca82cc174096 in term 0.
I20260928 02:38:55.887571 11752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1d49df507449e4bacb078234f77fb4" candidate_uuid: "c264570eeb9b42d8afa1ca82cc174096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "462a4a4221774bdfb2d2a129816b7fc7" is_pre_election: true
I20260928 02:38:55.887681 11752 raft_consensus.cc:2454] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c264570eeb9b42d8afa1ca82cc174096 in term 0.
I20260928 02:38:55.887697 11549 leader_election.cc:302] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f45b3e75f0441b8b351500acfab35f, c264570eeb9b42d8afa1ca82cc174096; no voters:
I20260928 02:38:55.887851 11971 raft_consensus.cc:2781] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:55.887894 11971 raft_consensus.cc:482] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:55.887913 11971 raft_consensus.cc:3037] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:55.888618 11971 raft_consensus.cc:504] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.888767 11971 leader_election.cc:288] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [CANDIDATE]: Term 1 election: Requested vote from peers 462a4a4221774bdfb2d2a129816b7fc7 (127.9.130.2:35417), 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471)
I20260928 02:38:55.888916 11752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1d49df507449e4bacb078234f77fb4" candidate_uuid: "c264570eeb9b42d8afa1ca82cc174096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "462a4a4221774bdfb2d2a129816b7fc7"
I20260928 02:38:55.888972 11896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1d49df507449e4bacb078234f77fb4" candidate_uuid: "c264570eeb9b42d8afa1ca82cc174096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f45b3e75f0441b8b351500acfab35f"
I20260928 02:38:55.889010 11752 raft_consensus.cc:3037] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:55.889034 11896 raft_consensus.cc:3037] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:55.889750 11896 raft_consensus.cc:2454] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c264570eeb9b42d8afa1ca82cc174096 in term 1.
I20260928 02:38:55.889751 11752 raft_consensus.cc:2454] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c264570eeb9b42d8afa1ca82cc174096 in term 1.
I20260928 02:38:55.889937 11547 leader_election.cc:302] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 462a4a4221774bdfb2d2a129816b7fc7, c264570eeb9b42d8afa1ca82cc174096; no voters:
I20260928 02:38:55.890048 11971 raft_consensus.cc:2781] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:55.890175 11971 raft_consensus.cc:686] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [term 1 LEADER]: Becoming Leader. State: Replica: c264570eeb9b42d8afa1ca82cc174096, State: Running, Role: LEADER
I20260928 02:38:55.890311 11971 consensus_queue.cc:237] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.890710 11989 tablet_replica.cc:439] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096: TxnStatusTablet state changed. Reason: New leader c264570eeb9b42d8afa1ca82cc174096. Latest consensus state: current_term: 1 leader_uuid: "c264570eeb9b42d8afa1ca82cc174096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } } }
I20260928 02:38:55.890795 11989 tablet_replica.cc:442] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:38:55.891057 11991 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:38:55.891218 11991 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:38:55.891115 11459 catalog_manager.cc:5495] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 reported cstate change: term changed from 0 to 1, leader changed from <none> to c264570eeb9b42d8afa1ca82cc174096 (127.9.130.1). New cstate: current_term: 1 leader_uuid: "c264570eeb9b42d8afa1ca82cc174096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } health_report { overall_health: UNKNOWN } } }
I20260928 02:38:55.897833 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.895109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.9.130.1:43053)
0928 02:38:55.895184 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.895186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.895208 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:38:55.895235 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.895236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.895253 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.895321 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:55.895415 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.896153 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.896913 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.897238 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.897309 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.897504 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:55.897615 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:55.897698 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.897721 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.897741 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:38:55.897802 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.895037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:43053 (local address 127.0.0.1:33954)
0928 02:38:55.895111 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.895124 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.895173 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.895329 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.895330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.895331 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:38:55.895371 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.895372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.896235 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.896236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.896793 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.896795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.897326 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.897327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.897390 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.897404 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:38:55.897682 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:38:55.897688 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.897716 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.897751 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:38:55.899576 11896 raft_consensus.cc:1261] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Refusing update from remote peer c264570eeb9b42d8afa1ca82cc174096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:55.899590 11752 raft_consensus.cc:1261] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Refusing update from remote peer c264570eeb9b42d8afa1ca82cc174096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:55.899852 11971 consensus_queue.cc:1035] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:55.900000 11989 consensus_queue.cc:1035] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:55.900915 11972 tablet_replica.cc:439] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7: TxnStatusTablet state changed. Reason: New leader c264570eeb9b42d8afa1ca82cc174096. Latest consensus state: current_term: 1 leader_uuid: "c264570eeb9b42d8afa1ca82cc174096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } } }
I20260928 02:38:55.901001 11972 tablet_replica.cc:442] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:38:55.901103 11973 tablet_replica.cc:439] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f: TxnStatusTablet state changed. Reason: New leader c264570eeb9b42d8afa1ca82cc174096. Latest consensus state: current_term: 1 leader_uuid: "c264570eeb9b42d8afa1ca82cc174096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } } }
I20260928 02:38:55.901196 11973 tablet_replica.cc:442] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:38:55.901980 11971 tablet_replica.cc:439] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c264570eeb9b42d8afa1ca82cc174096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } } }
I20260928 02:38:55.902061 11971 tablet_replica.cc:442] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:38:55.902185 11973 tablet_replica.cc:439] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c264570eeb9b42d8afa1ca82cc174096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } } }
I20260928 02:38:55.902266 11973 tablet_replica.cc:442] T 4f1d49df507449e4bacb078234f77fb4 P 13f45b3e75f0441b8b351500acfab35f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:38:55.902849 11971 tablet_replica.cc:439] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c264570eeb9b42d8afa1ca82cc174096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } } }
I20260928 02:38:55.902940 11971 tablet_replica.cc:442] T 4f1d49df507449e4bacb078234f77fb4 P c264570eeb9b42d8afa1ca82cc174096: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:38:55.903450 11995 mvcc.cc:205] Tried to move back new op lower bound from 7334146604641882112 to 7334146604606943232. Current Snapshot: MvccSnapshot[applied={T|T < 7334146604641882112}]
I20260928 02:38:55.903445 11972 tablet_replica.cc:439] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c264570eeb9b42d8afa1ca82cc174096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } } }
I20260928 02:38:55.903590 11972 tablet_replica.cc:442] T 4f1d49df507449e4bacb078234f77fb4 P 462a4a4221774bdfb2d2a129816b7fc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:38:55.907927 11988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.905255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.9.130.2:35417)
0928 02:38:55.905295 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.905297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.905364 (+ 67us) server_negotiation.cc:407] Connection header received
0928 02:38:55.905434 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.905435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.905449 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.905502 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:55.905621 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.906054 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.907149 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.907323 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.907381 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.907567 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:55.907651 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:55.907718 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.907778 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.907795 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:38:55.911419 11972 consensus_queue.cc:1035] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:55.913036 11972 consensus_queue.cc:1035] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:55.916594 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.913670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:39185 (local address 127.9.130.1:43053)
0928 02:38:55.913704 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.913706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.913898 (+ 192us) server_negotiation.cc:407] Connection header received
0928 02:38:55.913973 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.913974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.913982 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.914036 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:55.914310 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.914801 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.915927 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.916119 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.916171 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.916358 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:55.916396 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:55.916437 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.916520 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.916534 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:38:55.916669 12010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.913518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:43053 (local address 127.9.130.2:39185)
0928 02:38:55.913838 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.913848 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.913923 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.914188 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.914189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.914191 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:38:55.914248 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.914249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.914873 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.914874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.915841 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.915844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.916188 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.916189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.916253 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.916266 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:38:55.916450 (+ 184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:38:55.916451 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.916512 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.916557 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":74,"threads_started":1}
I20260928 02:38:55.925786 11988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.923091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:40543 (local address 127.9.130.2:35417)
0928 02:38:55.923132 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:55.923134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:55.923244 (+ 110us) server_negotiation.cc:407] Connection header received
0928 02:38:55.923320 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:55.923320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:55.923327 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:55.923380 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:55.923510 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.924004 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.925206 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.925399 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.925451 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.925621 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:55.925642 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:55.925673 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:55.925725 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:38:55.925740 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:38:55.925851 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:55.923051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:35417 (local address 127.9.130.1:40543)
0928 02:38:55.923186 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:55.923196 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:55.923264 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:55.923392 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:55.923393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:55.923394 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:38:55.923456 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.923457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.924070 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.924071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.925130 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:55.925132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:55.925469 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:55.925470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:55.925533 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:55.925547 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:38:55.925674 (+ 127us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:38:55.925676 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:38:55.925724 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:38:55.925760 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260928 02:38:55.935501 11458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:55.935621 11458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:55.936345 11458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554: Sending DeleteTablet for 3 replicas of tablet 8cbd79220b664f9f8f4aa1f3b4760d8c
I20260928 02:38:55.936667 11596 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbd79220b664f9f8f4aa1f3b4760d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:55 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:33940
I20260928 02:38:55.936738 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbd79220b664f9f8f4aa1f3b4760d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:55 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:44882
I20260928 02:38:55.937045 11736 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbd79220b664f9f8f4aa1f3b4760d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:55 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:59208
I20260928 02:38:55.937273 12017 tablet_replica.cc:330] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7: stopping tablet replica
I20260928 02:38:55.937268 11458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:38:55.937429 11458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:55.937621 12017 raft_consensus.cc:2229] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:38:55.937788 12017 raft_consensus.cc:2258] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:55.938402 12017 ts_tablet_manager.cc:1905] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:55.938633 11876 tablet_service.cc:1467] Processing CreateTablet for tablet 916e24deef134fa9b32afbd17d216da6 (DEFAULT_TABLE table=test-table [id=377cf2564898459da594bc8032027a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:55.938750 11876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916e24deef134fa9b32afbd17d216da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:55.940145 11966 tablet_bootstrap.cc:492] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f: Bootstrap starting.
I20260928 02:38:55.940378 12017 ts_tablet_manager.cc:1918] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:38:55.940495 12017 log.cc:1196] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/8cbd79220b664f9f8f4aa1f3b4760d8c
I20260928 02:38:55.940543 11966 tablet_bootstrap.cc:654] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:55.940891 12017 ts_tablet_manager.cc:1939] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 462a4a4221774bdfb2d2a129816b7fc7: Deleting consensus metadata
I20260928 02:38:55.941083 11596 tablet_service.cc:1467] Processing CreateTablet for tablet 916e24deef134fa9b32afbd17d216da6 (DEFAULT_TABLE table=test-table [id=377cf2564898459da594bc8032027a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:55.941193 11596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916e24deef134fa9b32afbd17d216da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:55.942539 11965 tablet_bootstrap.cc:492] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096: Bootstrap starting.
I20260928 02:38:55.942924 11965 tablet_bootstrap.cc:654] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:55.943641 11443 catalog_manager.cc:4841] TS 462a4a4221774bdfb2d2a129816b7fc7 (127.9.130.2:35417): tablet 8cbd79220b664f9f8f4aa1f3b4760d8c (table test-table [id=9fc65c69f3ea4718a276da477f962958]) successfully deleted
I20260928 02:38:55.943661 11736 tablet_service.cc:1467] Processing CreateTablet for tablet 916e24deef134fa9b32afbd17d216da6 (DEFAULT_TABLE table=test-table [id=377cf2564898459da594bc8032027a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:55.943746 11736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916e24deef134fa9b32afbd17d216da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:55.943908 12015 tablet_replica.cc:330] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096: stopping tablet replica
I20260928 02:38:55.944012 12015 raft_consensus.cc:2229] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:55.944073 12015 raft_consensus.cc:2258] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:55.944021 11965 tablet_bootstrap.cc:492] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096: No bootstrap required, opened a new log
I20260928 02:38:55.944205 12016 tablet_replica.cc:330] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f: stopping tablet replica
I20260928 02:38:55.944231 11965 ts_tablet_manager.cc:1397] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:38:55.944296 12016 raft_consensus.cc:2229] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:55.944345 12016 raft_consensus.cc:2258] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:55.944382 11965 raft_consensus.cc:348] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.944451 11965 raft_consensus.cc:374] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:55.944478 11965 raft_consensus.cc:729] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c264570eeb9b42d8afa1ca82cc174096, State: Initialized, Role: FOLLOWER
I20260928 02:38:55.944525 11965 consensus_queue.cc:260] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.944605 11965 ts_tablet_manager.cc:1428] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:55.944717 12015 ts_tablet_manager.cc:1905] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:55.944983 11967 tablet_bootstrap.cc:492] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7: Bootstrap starting.
I20260928 02:38:55.945362 11967 tablet_bootstrap.cc:654] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:55.945940 12016 ts_tablet_manager.cc:1905] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:55.946009 11967 tablet_bootstrap.cc:492] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7: No bootstrap required, opened a new log
I20260928 02:38:55.946058 11967 ts_tablet_manager.cc:1397] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:38:55.946193 11967 raft_consensus.cc:348] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.946276 11967 raft_consensus.cc:374] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:55.946308 11967 raft_consensus.cc:729] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 462a4a4221774bdfb2d2a129816b7fc7, State: Initialized, Role: FOLLOWER
I20260928 02:38:55.946367 12015 ts_tablet_manager.cc:1918] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:38:55.946355 11967 consensus_queue.cc:260] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.946431 12015 log.cc:1196] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/wal/wals/8cbd79220b664f9f8f4aa1f3b4760d8c
I20260928 02:38:55.946462 11967 ts_tablet_manager.cc:1428] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 02:38:55.946717 12015 ts_tablet_manager.cc:1939] T 8cbd79220b664f9f8f4aa1f3b4760d8c P c264570eeb9b42d8afa1ca82cc174096: Deleting consensus metadata
I20260928 02:38:55.947161 11443 catalog_manager.cc:4841] TS c264570eeb9b42d8afa1ca82cc174096 (127.9.130.1:43053): tablet 8cbd79220b664f9f8f4aa1f3b4760d8c (table test-table [id=9fc65c69f3ea4718a276da477f962958]) successfully deleted
I20260928 02:38:55.947907 12016 ts_tablet_manager.cc:1918] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:38:55.947991 12016 log.cc:1196] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/8cbd79220b664f9f8f4aa1f3b4760d8c
I20260928 02:38:55.948246 12016 ts_tablet_manager.cc:1939] T 8cbd79220b664f9f8f4aa1f3b4760d8c P 13f45b3e75f0441b8b351500acfab35f: Deleting consensus metadata
I20260928 02:38:55.948614 11445 catalog_manager.cc:4841] TS 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471): tablet 8cbd79220b664f9f8f4aa1f3b4760d8c (table test-table [id=9fc65c69f3ea4718a276da477f962958]) successfully deleted
I20260928 02:38:55.948832 11966 tablet_bootstrap.cc:492] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f: No bootstrap required, opened a new log
I20260928 02:38:55.948892 11966 ts_tablet_manager.cc:1397] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260928 02:38:55.949025 11966 raft_consensus.cc:348] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.949095 11966 raft_consensus.cc:374] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:55.949122 11966 raft_consensus.cc:729] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f45b3e75f0441b8b351500acfab35f, State: Initialized, Role: FOLLOWER
I20260928 02:38:55.949164 11966 consensus_queue.cc:260] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:55.949249 11966 ts_tablet_manager.cc:1428] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:56.179364 12007 raft_consensus.cc:482] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:56.179472 12007 raft_consensus.cc:504] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:56.179636 12007 leader_election.cc:288] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c264570eeb9b42d8afa1ca82cc174096 (127.9.130.1:43053), 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471)
I20260928 02:38:56.179873 11896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916e24deef134fa9b32afbd17d216da6" candidate_uuid: "462a4a4221774bdfb2d2a129816b7fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f45b3e75f0441b8b351500acfab35f" is_pre_election: true
I20260928 02:38:56.179951 11896 raft_consensus.cc:2454] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 462a4a4221774bdfb2d2a129816b7fc7 in term 0.
I20260928 02:38:56.179884 11616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916e24deef134fa9b32afbd17d216da6" candidate_uuid: "462a4a4221774bdfb2d2a129816b7fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c264570eeb9b42d8afa1ca82cc174096" is_pre_election: true
I20260928 02:38:56.180034 11616 raft_consensus.cc:2454] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 462a4a4221774bdfb2d2a129816b7fc7 in term 0.
I20260928 02:38:56.180136 11689 leader_election.cc:302] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f45b3e75f0441b8b351500acfab35f, 462a4a4221774bdfb2d2a129816b7fc7; no voters:
I20260928 02:38:56.180255 12007 raft_consensus.cc:2781] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:56.180310 12007 raft_consensus.cc:482] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:56.180341 12007 raft_consensus.cc:3037] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:56.181166 12007 raft_consensus.cc:504] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:56.181319 12007 leader_election.cc:288] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [CANDIDATE]: Term 1 election: Requested vote from peers c264570eeb9b42d8afa1ca82cc174096 (127.9.130.1:43053), 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471)
I20260928 02:38:56.181505 11896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916e24deef134fa9b32afbd17d216da6" candidate_uuid: "462a4a4221774bdfb2d2a129816b7fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f45b3e75f0441b8b351500acfab35f"
I20260928 02:38:56.181505 11616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916e24deef134fa9b32afbd17d216da6" candidate_uuid: "462a4a4221774bdfb2d2a129816b7fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c264570eeb9b42d8afa1ca82cc174096"
I20260928 02:38:56.181574 11616 raft_consensus.cc:3037] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:56.181574 11896 raft_consensus.cc:3037] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:56.182343 11616 raft_consensus.cc:2454] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 462a4a4221774bdfb2d2a129816b7fc7 in term 1.
I20260928 02:38:56.182343 11896 raft_consensus.cc:2454] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 462a4a4221774bdfb2d2a129816b7fc7 in term 1.
I20260928 02:38:56.182520 11687 leader_election.cc:302] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 462a4a4221774bdfb2d2a129816b7fc7, c264570eeb9b42d8afa1ca82cc174096; no voters:
I20260928 02:38:56.182646 12007 raft_consensus.cc:2781] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:56.182700 12007 raft_consensus.cc:686] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 462a4a4221774bdfb2d2a129816b7fc7, State: Running, Role: LEADER
I20260928 02:38:56.182766 12007 consensus_queue.cc:237] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:56.183333 11458 catalog_manager.cc:5495] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 462a4a4221774bdfb2d2a129816b7fc7 (127.9.130.2). New cstate: current_term: 1 leader_uuid: "462a4a4221774bdfb2d2a129816b7fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } health_report { overall_health: UNKNOWN } } }
I20260928 02:38:56.233500 11896 raft_consensus.cc:1261] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Refusing update from remote peer 462a4a4221774bdfb2d2a129816b7fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:56.233500 11616 raft_consensus.cc:1261] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Refusing update from remote peer 462a4a4221774bdfb2d2a129816b7fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:56.233690 12007 consensus_queue.cc:1035] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:56.233808 11975 consensus_queue.cc:1035] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:56.235044 11996 mvcc.cc:205] Tried to move back new op lower bound from 7334146606011183104 to 7334146605804904448. Current Snapshot: MvccSnapshot[applied={T|T < 7334146606011183104}]
W20260928 02:38:56.238188 11995 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:38:57.754807 11459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:57.754901 11459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:57.755708 11459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554: Sending DeleteTablet for 3 replicas of tablet 916e24deef134fa9b32afbd17d216da6
I20260928 02:38:57.755983 11596 tablet_service.cc:1514] Processing DeleteTablet for tablet 916e24deef134fa9b32afbd17d216da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:57 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:33940
I20260928 02:38:57.756006 11736 tablet_service.cc:1514] Processing DeleteTablet for tablet 916e24deef134fa9b32afbd17d216da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:57 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:59208
I20260928 02:38:57.756197 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 916e24deef134fa9b32afbd17d216da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:57 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:44882
I20260928 02:38:57.756554 12046 tablet_replica.cc:330] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7: stopping tablet replica
I20260928 02:38:57.756611 12046 raft_consensus.cc:2229] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:38:57.756664 12046 raft_consensus.cc:2258] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:57.756875 12046 ts_tablet_manager.cc:1905] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:57.758699 12046 ts_tablet_manager.cc:1918] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:38:57.758769 12046 log.cc:1196] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/916e24deef134fa9b32afbd17d216da6
I20260928 02:38:57.759020 12046 ts_tablet_manager.cc:1939] T 916e24deef134fa9b32afbd17d216da6 P 462a4a4221774bdfb2d2a129816b7fc7: Deleting consensus metadata
I20260928 02:38:57.759186 12047 tablet_replica.cc:330] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f: stopping tablet replica
I20260928 02:38:57.759246 12047 raft_consensus.cc:2229] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:57.759298 12047 raft_consensus.cc:2258] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:57.759399 12045 tablet_replica.cc:330] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096: stopping tablet replica
I20260928 02:38:57.759539 11443 catalog_manager.cc:4841] TS 462a4a4221774bdfb2d2a129816b7fc7 (127.9.130.2:35417): tablet 916e24deef134fa9b32afbd17d216da6 (table test-table [id=377cf2564898459da594bc8032027a1a]) successfully deleted
I20260928 02:38:57.759548 12045 raft_consensus.cc:2229] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:57.759608 12045 raft_consensus.cc:2258] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:57.759601 12047 ts_tablet_manager.cc:1905] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:57.759805 12045 ts_tablet_manager.cc:1905] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:57.761250 12047 ts_tablet_manager.cc:1918] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:38:57.761339 12047 log.cc:1196] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/916e24deef134fa9b32afbd17d216da6
I20260928 02:38:57.761396 12045 ts_tablet_manager.cc:1918] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:38:57.761456 12045 log.cc:1196] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/wal/wals/916e24deef134fa9b32afbd17d216da6
I20260928 02:38:57.761575 12047 ts_tablet_manager.cc:1939] T 916e24deef134fa9b32afbd17d216da6 P 13f45b3e75f0441b8b351500acfab35f: Deleting consensus metadata
I20260928 02:38:57.761689 12045 ts_tablet_manager.cc:1939] T 916e24deef134fa9b32afbd17d216da6 P c264570eeb9b42d8afa1ca82cc174096: Deleting consensus metadata
I20260928 02:38:57.762279 11445 catalog_manager.cc:4841] TS 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471): tablet 916e24deef134fa9b32afbd17d216da6 (table test-table [id=377cf2564898459da594bc8032027a1a]) successfully deleted
I20260928 02:38:57.762354 11443 catalog_manager.cc:4841] TS c264570eeb9b42d8afa1ca82cc174096 (127.9.130.1:43053): tablet 916e24deef134fa9b32afbd17d216da6 (table test-table [id=377cf2564898459da594bc8032027a1a]) successfully deleted
Sep 28 02:38:57 dist-test-slave-kkrh krb5kdc[11414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563137, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:38:57 dist-test-slave-kkrh krb5kdc[11414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563137, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:38:57.780052 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:57.772917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.9.130.62:46531)
0928 02:38:57.773096 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:57.773097 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:57.773106 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:57.773183 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:57.773183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:57.773192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:57.773253 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:57.773828 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:57.774360 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:57.775104 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:57.775297 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:57.775353 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:57.777680 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:57.777681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:57.777682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:57.777687 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:57.778991 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:57.779320 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:57.779321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:57.779321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:57.779374 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:57.779538 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:57.779538 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:57.779539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:57.779637 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:57.779709 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:57.779794 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:38:57.779959 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":134,"threads_started":1}
I20260928 02:38:57.781492 11459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:38:57.781656 11459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:57.782987 11596 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0c742d32b543c6b2c04070c03fbd46 (DEFAULT_TABLE table=test-table [id=c94a1d27405b4997912781c9819acc6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:57.783010 11736 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0c742d32b543c6b2c04070c03fbd46 (DEFAULT_TABLE table=test-table [id=c94a1d27405b4997912781c9819acc6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:57.783154 11596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0c742d32b543c6b2c04070c03fbd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:57.783174 11736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0c742d32b543c6b2c04070c03fbd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:57.784330 12056 tablet_bootstrap.cc:492] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7: Bootstrap starting.
I20260928 02:38:57.784315 11876 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0c742d32b543c6b2c04070c03fbd46 (DEFAULT_TABLE table=test-table [id=c94a1d27405b4997912781c9819acc6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:57.784423 11876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0c742d32b543c6b2c04070c03fbd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:57.784683 12056 tablet_bootstrap.cc:654] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:57.785233 12056 tablet_bootstrap.cc:492] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7: No bootstrap required, opened a new log
I20260928 02:38:57.785288 12056 ts_tablet_manager.cc:1397] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:57.785390 12056 raft_consensus.cc:348] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.785459 12056 raft_consensus.cc:374] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:57.785478 12056 raft_consensus.cc:729] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 462a4a4221774bdfb2d2a129816b7fc7, State: Initialized, Role: FOLLOWER
I20260928 02:38:57.785511 12056 consensus_queue.cc:260] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.785626 12056 ts_tablet_manager.cc:1428] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:57.785769 12059 tablet_bootstrap.cc:492] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f: Bootstrap starting.
I20260928 02:38:57.785917 12057 tablet_bootstrap.cc:492] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096: Bootstrap starting.
I20260928 02:38:57.786316 12057 tablet_bootstrap.cc:654] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:57.786329 12059 tablet_bootstrap.cc:654] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:57.786934 12057 tablet_bootstrap.cc:492] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096: No bootstrap required, opened a new log
I20260928 02:38:57.786990 12057 ts_tablet_manager.cc:1397] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:57.787000 12059 tablet_bootstrap.cc:492] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f: No bootstrap required, opened a new log
I20260928 02:38:57.787039 12059 ts_tablet_manager.cc:1397] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:57.787124 12057 raft_consensus.cc:348] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.787201 12057 raft_consensus.cc:374] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:57.787171 12059 raft_consensus.cc:348] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.787222 12057 raft_consensus.cc:729] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c264570eeb9b42d8afa1ca82cc174096, State: Initialized, Role: FOLLOWER
I20260928 02:38:57.787236 12059 raft_consensus.cc:374] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:57.787271 12059 raft_consensus.cc:729] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f45b3e75f0441b8b351500acfab35f, State: Initialized, Role: FOLLOWER
I20260928 02:38:57.787268 12057 consensus_queue.cc:260] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.787317 12059 consensus_queue.cc:260] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.787372 12057 ts_tablet_manager.cc:1428] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:57.787470 12059 ts_tablet_manager.cc:1428] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:57.870900 12039 raft_consensus.cc:482] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:57.870994 12039 raft_consensus.cc:504] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.871179 12039 leader_election.cc:288] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 462a4a4221774bdfb2d2a129816b7fc7 (127.9.130.2:35417), 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471)
I20260928 02:38:57.871413 11752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0c742d32b543c6b2c04070c03fbd46" candidate_uuid: "c264570eeb9b42d8afa1ca82cc174096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "462a4a4221774bdfb2d2a129816b7fc7" is_pre_election: true
I20260928 02:38:57.871405 11896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0c742d32b543c6b2c04070c03fbd46" candidate_uuid: "c264570eeb9b42d8afa1ca82cc174096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f45b3e75f0441b8b351500acfab35f" is_pre_election: true
I20260928 02:38:57.871500 11752 raft_consensus.cc:2454] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c264570eeb9b42d8afa1ca82cc174096 in term 0.
I20260928 02:38:57.871522 11896 raft_consensus.cc:2454] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c264570eeb9b42d8afa1ca82cc174096 in term 0.
I20260928 02:38:57.871668 11547 leader_election.cc:302] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 462a4a4221774bdfb2d2a129816b7fc7, c264570eeb9b42d8afa1ca82cc174096; no voters:
I20260928 02:38:57.871827 12039 raft_consensus.cc:2781] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:57.871884 12039 raft_consensus.cc:482] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:57.871909 12039 raft_consensus.cc:3037] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:57.872685 12039 raft_consensus.cc:504] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.872872 12039 leader_election.cc:288] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [CANDIDATE]: Term 1 election: Requested vote from peers 462a4a4221774bdfb2d2a129816b7fc7 (127.9.130.2:35417), 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471)
I20260928 02:38:57.873008 11896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0c742d32b543c6b2c04070c03fbd46" candidate_uuid: "c264570eeb9b42d8afa1ca82cc174096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f45b3e75f0441b8b351500acfab35f"
I20260928 02:38:57.873096 11896 raft_consensus.cc:3037] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:57.873083 11752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0c742d32b543c6b2c04070c03fbd46" candidate_uuid: "c264570eeb9b42d8afa1ca82cc174096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "462a4a4221774bdfb2d2a129816b7fc7"
I20260928 02:38:57.873145 11752 raft_consensus.cc:3037] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:57.873862 11896 raft_consensus.cc:2454] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c264570eeb9b42d8afa1ca82cc174096 in term 1.
I20260928 02:38:57.873872 11752 raft_consensus.cc:2454] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c264570eeb9b42d8afa1ca82cc174096 in term 1.
I20260928 02:38:57.874087 11549 leader_election.cc:302] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f45b3e75f0441b8b351500acfab35f, c264570eeb9b42d8afa1ca82cc174096; no voters:
I20260928 02:38:57.874248 12039 raft_consensus.cc:2781] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:57.874325 12039 raft_consensus.cc:686] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [term 1 LEADER]: Becoming Leader. State: Replica: c264570eeb9b42d8afa1ca82cc174096, State: Running, Role: LEADER
I20260928 02:38:57.874378 12039 consensus_queue.cc:237] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.875123 11459 catalog_manager.cc:5495] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 reported cstate change: term changed from 0 to 1, leader changed from <none> to c264570eeb9b42d8afa1ca82cc174096 (127.9.130.1). New cstate: current_term: 1 leader_uuid: "c264570eeb9b42d8afa1ca82cc174096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } health_report { overall_health: UNKNOWN } } }
I20260928 02:38:57.904422 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:57.901914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33968 (local address 127.9.130.1:43053)
0928 02:38:57.902090 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:57.902092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:57.902101 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:57.902125 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:57.902126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:57.902134 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:57.902208 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:57.902355 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:57.902841 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:57.903617 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:57.903886 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:57.903962 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:57.904177 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:57.904231 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:57.904291 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:57.904342 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 02:38:57.904364 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":109,"threads_started":1}
I20260928 02:38:57.908804 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:57.906497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:44085 (local address 127.9.130.1:43053)
0928 02:38:57.906561 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:57.906563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:57.906635 (+ 72us) server_negotiation.cc:407] Connection header received
0928 02:38:57.906706 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:57.906706 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:57.906715 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:57.906766 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:57.906873 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:57.907323 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:57.908140 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:57.908350 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:57.908414 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:57.908567 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:57.908611 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:57.908652 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:57.908716 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 02:38:57.908736 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":46}
I20260928 02:38:57.908871 12068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:57.906429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:43053 (local address 127.9.130.1:44085)
0928 02:38:57.906583 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:57.906592 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:57.906650 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:57.906769 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:57.906770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:57.906771 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:38:57.906817 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:57.906818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:57.907417 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:57.907418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:57.908051 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:57.908053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:57.908422 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:57.908423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:57.908496 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:57.908500 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:38:57.908676 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:38:57.908677 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:38:57.908714 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 02:38:57.908758 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":7,"thread_start_us":76,"threads_started":1}
I20260928 02:38:57.912523 11752 raft_consensus.cc:1261] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Refusing update from remote peer c264570eeb9b42d8afa1ca82cc174096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:57.912537 11896 raft_consensus.cc:1261] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Refusing update from remote peer c264570eeb9b42d8afa1ca82cc174096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:57.912740 12026 consensus_queue.cc:1035] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:57.912842 12039 consensus_queue.cc:1035] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:57.913826 12024 mvcc.cc:205] Tried to move back new op lower bound from 7334146612888186880 to 7334146612733620224. Current Snapshot: MvccSnapshot[applied={T|T < 7334146612888186880}]
I20260928 02:38:57.922839 11458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:57.922930 11458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:57.923545 11458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554: Sending DeleteTablet for 3 replicas of tablet 4e0c742d32b543c6b2c04070c03fbd46
I20260928 02:38:57.923786 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e0c742d32b543c6b2c04070c03fbd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:57 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:44882
I20260928 02:38:57.923790 11736 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e0c742d32b543c6b2c04070c03fbd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:57 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:59208
I20260928 02:38:57.923895 12047 tablet_replica.cc:330] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f: stopping tablet replica
I20260928 02:38:57.923905 12046 tablet_replica.cc:330] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7: stopping tablet replica
I20260928 02:38:57.923939 12047 raft_consensus.cc:2229] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:57.923951 12046 raft_consensus.cc:2229] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:57.923972 12047 raft_consensus.cc:2258] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:57.923981 12046 raft_consensus.cc:2258] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:57.924105 11596 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e0c742d32b543c6b2c04070c03fbd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:57 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:33940
I20260928 02:38:57.924180 12045 tablet_replica.cc:330] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096: stopping tablet replica
I20260928 02:38:57.924227 12045 raft_consensus.cc:2229] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:38:57.924250 12047 ts_tablet_manager.cc:1905] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:57.924299 12045 raft_consensus.cc:2258] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:57.924453 12046 ts_tablet_manager.cc:1905] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:57.924620 12045 ts_tablet_manager.cc:1905] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:57.925844 12047 ts_tablet_manager.cc:1918] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:38:57.925844 12046 ts_tablet_manager.cc:1918] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:38:57.925894 12046 log.cc:1196] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/4e0c742d32b543c6b2c04070c03fbd46
I20260928 02:38:57.925894 12047 log.cc:1196] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/4e0c742d32b543c6b2c04070c03fbd46
I20260928 02:38:57.926052 12045 ts_tablet_manager.cc:1918] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:38:57.926103 12045 log.cc:1196] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-0/wal/wals/4e0c742d32b543c6b2c04070c03fbd46
I20260928 02:38:57.926113 12047 ts_tablet_manager.cc:1939] T 4e0c742d32b543c6b2c04070c03fbd46 P 13f45b3e75f0441b8b351500acfab35f: Deleting consensus metadata
I20260928 02:38:57.926126 12046 ts_tablet_manager.cc:1939] T 4e0c742d32b543c6b2c04070c03fbd46 P 462a4a4221774bdfb2d2a129816b7fc7: Deleting consensus metadata
I20260928 02:38:57.926313 12045 ts_tablet_manager.cc:1939] T 4e0c742d32b543c6b2c04070c03fbd46 P c264570eeb9b42d8afa1ca82cc174096: Deleting consensus metadata
I20260928 02:38:57.926498 11445 catalog_manager.cc:4841] TS 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471): tablet 4e0c742d32b543c6b2c04070c03fbd46 (table test-table [id=c94a1d27405b4997912781c9819acc6c]) successfully deleted
I20260928 02:38:57.926764 11443 catalog_manager.cc:4841] TS c264570eeb9b42d8afa1ca82cc174096 (127.9.130.1:43053): tablet 4e0c742d32b543c6b2c04070c03fbd46 (table test-table [id=c94a1d27405b4997912781c9819acc6c]) successfully deleted
I20260928 02:38:57.926833 11443 catalog_manager.cc:4841] TS 462a4a4221774bdfb2d2a129816b7fc7 (127.9.130.2:35417): tablet 4e0c742d32b543c6b2c04070c03fbd46 (table test-table [id=c94a1d27405b4997912781c9819acc6c]) successfully deleted
I20260928 02:38:57.926930 11458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:38:57.927073 11458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:57.928336 11596 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae2b5bdd0da4eeb93a1c280144ab8de (DEFAULT_TABLE table=test-table [id=a82422e55050478e82df44157fcceebd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:57.928354 11736 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae2b5bdd0da4eeb93a1c280144ab8de (DEFAULT_TABLE table=test-table [id=a82422e55050478e82df44157fcceebd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:57.928442 11596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae2b5bdd0da4eeb93a1c280144ab8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:57.928474 11736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae2b5bdd0da4eeb93a1c280144ab8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:57.929270 12056 tablet_bootstrap.cc:492] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7: Bootstrap starting.
I20260928 02:38:57.929370 11876 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae2b5bdd0da4eeb93a1c280144ab8de (DEFAULT_TABLE table=test-table [id=a82422e55050478e82df44157fcceebd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:57.929464 11876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae2b5bdd0da4eeb93a1c280144ab8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:57.929557 12056 tablet_bootstrap.cc:654] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:57.929555 12057 tablet_bootstrap.cc:492] T 4ae2b5bdd0da4eeb93a1c280144ab8de P c264570eeb9b42d8afa1ca82cc174096: Bootstrap starting.
I20260928 02:38:57.929867 12057 tablet_bootstrap.cc:654] T 4ae2b5bdd0da4eeb93a1c280144ab8de P c264570eeb9b42d8afa1ca82cc174096: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:57.930477 12057 tablet_bootstrap.cc:492] T 4ae2b5bdd0da4eeb93a1c280144ab8de P c264570eeb9b42d8afa1ca82cc174096: No bootstrap required, opened a new log
I20260928 02:38:57.930526 12057 ts_tablet_manager.cc:1397] T 4ae2b5bdd0da4eeb93a1c280144ab8de P c264570eeb9b42d8afa1ca82cc174096: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:57.930620 12057 raft_consensus.cc:348] T 4ae2b5bdd0da4eeb93a1c280144ab8de P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.930692 12057 raft_consensus.cc:374] T 4ae2b5bdd0da4eeb93a1c280144ab8de P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:57.930712 12057 raft_consensus.cc:729] T 4ae2b5bdd0da4eeb93a1c280144ab8de P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c264570eeb9b42d8afa1ca82cc174096, State: Initialized, Role: FOLLOWER
I20260928 02:38:57.930727 12059 tablet_bootstrap.cc:492] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f: Bootstrap starting.
I20260928 02:38:57.930751 12057 consensus_queue.cc:260] T 4ae2b5bdd0da4eeb93a1c280144ab8de P c264570eeb9b42d8afa1ca82cc174096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.930827 12057 ts_tablet_manager.cc:1428] T 4ae2b5bdd0da4eeb93a1c280144ab8de P c264570eeb9b42d8afa1ca82cc174096: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:57.931272 12059 tablet_bootstrap.cc:654] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:57.931728 12056 tablet_bootstrap.cc:492] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7: No bootstrap required, opened a new log
I20260928 02:38:57.931767 12056 ts_tablet_manager.cc:1397] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:38:57.931921 12059 tablet_bootstrap.cc:492] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f: No bootstrap required, opened a new log
I20260928 02:38:57.931905 12056 raft_consensus.cc:348] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.931983 12056 raft_consensus.cc:374] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:57.931984 12059 ts_tablet_manager.cc:1397] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:57.932004 12056 raft_consensus.cc:729] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 462a4a4221774bdfb2d2a129816b7fc7, State: Initialized, Role: FOLLOWER
I20260928 02:38:57.932050 12056 consensus_queue.cc:260] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.932139 12056 ts_tablet_manager.cc:1428] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:57.932114 12059 raft_consensus.cc:348] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.932171 12059 raft_consensus.cc:374] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:57.932191 12059 raft_consensus.cc:729] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f45b3e75f0441b8b351500acfab35f, State: Initialized, Role: FOLLOWER
I20260928 02:38:57.932229 12059 consensus_queue.cc:260] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.932301 12059 ts_tablet_manager.cc:1428] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:57.940323 11972 raft_consensus.cc:482] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:57.940404 11972 raft_consensus.cc:504] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.940559 11972 leader_election.cc:288] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c264570eeb9b42d8afa1ca82cc174096 (127.9.130.1:43053), 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471)
I20260928 02:38:57.940711 11896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae2b5bdd0da4eeb93a1c280144ab8de" candidate_uuid: "462a4a4221774bdfb2d2a129816b7fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f45b3e75f0441b8b351500acfab35f" is_pre_election: true
I20260928 02:38:57.940716 11616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae2b5bdd0da4eeb93a1c280144ab8de" candidate_uuid: "462a4a4221774bdfb2d2a129816b7fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c264570eeb9b42d8afa1ca82cc174096" is_pre_election: true
I20260928 02:38:57.940799 11896 raft_consensus.cc:2454] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 462a4a4221774bdfb2d2a129816b7fc7 in term 0.
I20260928 02:38:57.940816 11616 raft_consensus.cc:2454] T 4ae2b5bdd0da4eeb93a1c280144ab8de P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 462a4a4221774bdfb2d2a129816b7fc7 in term 0.
I20260928 02:38:57.940924 11689 leader_election.cc:302] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f45b3e75f0441b8b351500acfab35f, 462a4a4221774bdfb2d2a129816b7fc7; no voters:
I20260928 02:38:57.941020 11972 raft_consensus.cc:2781] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:57.941056 11972 raft_consensus.cc:482] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:57.941075 11972 raft_consensus.cc:3037] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:57.941828 11972 raft_consensus.cc:504] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.941955 11972 leader_election.cc:288] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [CANDIDATE]: Term 1 election: Requested vote from peers c264570eeb9b42d8afa1ca82cc174096 (127.9.130.1:43053), 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471)
I20260928 02:38:57.942134 11896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae2b5bdd0da4eeb93a1c280144ab8de" candidate_uuid: "462a4a4221774bdfb2d2a129816b7fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f45b3e75f0441b8b351500acfab35f"
I20260928 02:38:57.942083 11616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae2b5bdd0da4eeb93a1c280144ab8de" candidate_uuid: "462a4a4221774bdfb2d2a129816b7fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c264570eeb9b42d8afa1ca82cc174096"
I20260928 02:38:57.942204 11896 raft_consensus.cc:3037] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:57.942216 11616 raft_consensus.cc:3037] T 4ae2b5bdd0da4eeb93a1c280144ab8de P c264570eeb9b42d8afa1ca82cc174096 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:57.942880 11896 raft_consensus.cc:2454] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 462a4a4221774bdfb2d2a129816b7fc7 in term 1.
I20260928 02:38:57.942880 11616 raft_consensus.cc:2454] T 4ae2b5bdd0da4eeb93a1c280144ab8de P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 462a4a4221774bdfb2d2a129816b7fc7 in term 1.
I20260928 02:38:57.943054 11689 leader_election.cc:302] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f45b3e75f0441b8b351500acfab35f, 462a4a4221774bdfb2d2a129816b7fc7; no voters:
I20260928 02:38:57.943167 11972 raft_consensus.cc:2781] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:57.943233 11972 raft_consensus.cc:686] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 462a4a4221774bdfb2d2a129816b7fc7, State: Running, Role: LEADER
I20260928 02:38:57.943279 11972 consensus_queue.cc:237] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } }
I20260928 02:38:57.943785 11459 catalog_manager.cc:5495] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 462a4a4221774bdfb2d2a129816b7fc7 (127.9.130.2). New cstate: current_term: 1 leader_uuid: "462a4a4221774bdfb2d2a129816b7fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "462a4a4221774bdfb2d2a129816b7fc7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 } health_report { overall_health: UNKNOWN } } }
I20260928 02:38:57.951442 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:57.949074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.9.130.2:35417)
0928 02:38:57.949244 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:57.949246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:57.949251 (+ 5us) server_negotiation.cc:407] Connection header received
0928 02:38:57.949268 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:57.949269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:57.949277 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:57.949350 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:57.949454 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:57.949941 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:57.950720 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:57.950936 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:57.951013 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:57.951150 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:57.951192 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:57.951253 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:57.951312 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:38:57.951333 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260928 02:38:57.953706 11896 raft_consensus.cc:1261] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Refusing update from remote peer 462a4a4221774bdfb2d2a129816b7fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:57.953743 11616 raft_consensus.cc:1261] T 4ae2b5bdd0da4eeb93a1c280144ab8de P c264570eeb9b42d8afa1ca82cc174096 [term 1 FOLLOWER]: Refusing update from remote peer 462a4a4221774bdfb2d2a129816b7fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:57.953918 11972 consensus_queue.cc:1035] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c264570eeb9b42d8afa1ca82cc174096" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:57.954006 12040 consensus_queue.cc:1035] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13f45b3e75f0441b8b351500acfab35f" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:38:57.958081 12028 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:38:59.478008 11458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:59.478108 11458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:59.478948 11458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85b798dca7a54130b549933e8d9c4554: Sending DeleteTablet for 3 replicas of tablet 4ae2b5bdd0da4eeb93a1c280144ab8de
I20260928 02:38:59.479197 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae2b5bdd0da4eeb93a1c280144ab8de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:59 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:44882
I20260928 02:38:59.479226 11736 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae2b5bdd0da4eeb93a1c280144ab8de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:59 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:59208
I20260928 02:38:59.479267 11596 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae2b5bdd0da4eeb93a1c280144ab8de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:59 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:33940
I20260928 02:38:59.479540 12101 tablet_replica.cc:330] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f: stopping tablet replica
I20260928 02:38:59.479579 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 11531
I20260928 02:38:59.479633 12101 raft_consensus.cc:2229] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:59.481407 12101 raft_consensus.cc:2258] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:59.481683 12101 ts_tablet_manager.cc:1905] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:59.483404 12101 ts_tablet_manager.cc:1918] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:38:59.483503 12101 log.cc:1196] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/4ae2b5bdd0da4eeb93a1c280144ab8de
I20260928 02:38:59.483713 12101 ts_tablet_manager.cc:1939] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 13f45b3e75f0441b8b351500acfab35f: Deleting consensus metadata
I20260928 02:38:59.484066 11445 catalog_manager.cc:4841] TS 13f45b3e75f0441b8b351500acfab35f (127.9.130.3:39471): tablet 4ae2b5bdd0da4eeb93a1c280144ab8de (table test-table [id=a82422e55050478e82df44157fcceebd]) successfully deleted
I20260928 02:38:59.485673 12102 tablet_replica.cc:330] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7: stopping tablet replica
I20260928 02:38:59.485751 12102 raft_consensus.cc:2229] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:38:59.485842 12102 raft_consensus.cc:2258] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:59.486096 12102 ts_tablet_manager.cc:1905] T 4ae2b5bdd0da4eeb93a1c280144ab8de P 462a4a4221774bdfb2d2a129816b7fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:38:59.486193 11443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:43053 (error 108)
I20260928 02:38:59.486413 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 11671
I20260928 02:38:59.487929 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:59.486561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:43053 (local address 127.0.0.1:33974)
0928 02:38:59.487830 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:59.487879 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:43053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1229,"thread_start_us":60,"threads_started":1}
W20260928 02:38:59.488037 11443 catalog_manager.cc:4584] TS c264570eeb9b42d8afa1ca82cc174096 (127.9.130.1:43053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ae2b5bdd0da4eeb93a1c280144ab8de: Network error: Client connection negotiation failed: client connection to 127.9.130.1:43053: connect: Connection refused (error 111)
I20260928 02:38:59.492010 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 11811
I20260928 02:38:59.492066 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:59.491951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:35417 (local address 127.0.0.1:59236)
0928 02:38:59.492004 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:59.492034 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.2:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:38:59.496590 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 11427
2026-09-28T02:38:59Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6047 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:59 dist-test-slave-kkrh krb5kdc[12108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:59 dist-test-slave-kkrh krb5kdc[12108](info): set up 2 sockets
Sep 28 02:38:59 dist-test-slave-kkrh krb5kdc[12108](info): commencing operation
krb5kdc: starting...
W20260928 02:39:00.531994 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:39:00 dist-test-slave-kkrh krb5kdc[12108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:39:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:39:00.646417 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:38099
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:38099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:00.725051 12121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:00.725273 12121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:00.725309 12121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:00.726449 12121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:00.726521 12121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:00.726547 12121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:00.726575 12121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:00.726598 12121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:00.728133 12121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:38099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:38099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:00.728395 12121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:00.728608 12121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:00.733387 12127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:00.733392 12126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:00.733536 12121 server_base.cc:1034] running on GCE node
W20260928 02:39:00.733486 12129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:00.733861 12121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:00.734146 12121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:00.735347 12121 hybrid_clock.cc:648] HybridClock initialized: now 1790563140735316 us; error 49 us; skew 500 ppm
Sep 28 02:39:00 dist-test-slave-kkrh krb5kdc[12108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563140, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:00.740878 12121 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:39:00.741307 12121 webserver.cc:460] Webserver started at http://127.9.130.62:39077/ using document root <none> and password file <none>
I20260928 02:39:00.741489 12121 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:00.741528 12121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:00.741613 12121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:00.742599 12121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "5c39374a36b24e2da8e814e842204ba3"
format_stamp: "Formatted at 2026-09-28 02:39:00 on dist-test-slave-kkrh"
server_key: "ae3429f96b10d3507513288a22a1745c"
server_key_iv: "1d82fafaada42acd381cac4fedfdd77d"
server_key_version: "encryptionkey@0"
I20260928 02:39:00.742909 12121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "5c39374a36b24e2da8e814e842204ba3"
format_stamp: "Formatted at 2026-09-28 02:39:00 on dist-test-slave-kkrh"
server_key: "ae3429f96b10d3507513288a22a1745c"
server_key_iv: "1d82fafaada42acd381cac4fedfdd77d"
server_key_version: "encryptionkey@0"
I20260928 02:39:00.744208 12121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:39:00.744875 12136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:00.745072 12121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:00.745153 12121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "5c39374a36b24e2da8e814e842204ba3"
format_stamp: "Formatted at 2026-09-28 02:39:00 on dist-test-slave-kkrh"
server_key: "ae3429f96b10d3507513288a22a1745c"
server_key_iv: "1d82fafaada42acd381cac4fedfdd77d"
server_key_version: "encryptionkey@0"
I20260928 02:39:00.745204 12121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:00.762339 12121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:00.764060 12121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:00.764163 12121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:00.767582 12121 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:38099
I20260928 02:39:00.767637 12188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:38099 every 8 connection(s)
I20260928 02:39:00.768030 12121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:00.768904 12189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:00.770759 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 12121
I20260928 02:39:00.770833 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:00.771001 9736 external_mini_cluster.cc:1428] Setting key 841e03d3413af97a5f3902a0088b5e76
I20260928 02:39:00.771102 12189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3: Bootstrap starting.
I20260928 02:39:00.771800 12189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:00.772126 12189 log.cc:824] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:00.772761 12189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3: No bootstrap required, opened a new log
I20260928 02:39:00.774041 12189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c39374a36b24e2da8e814e842204ba3" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38099 } }
I20260928 02:39:00.774190 12189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:00.774236 12189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c39374a36b24e2da8e814e842204ba3, State: Initialized, Role: FOLLOWER
I20260928 02:39:00.774353 12189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c39374a36b24e2da8e814e842204ba3" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38099 } }
I20260928 02:39:00.774446 12189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:00.774495 12189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:00.774542 12189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:00.775081 12189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c39374a36b24e2da8e814e842204ba3" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38099 } }
I20260928 02:39:00.775203 12189 leader_election.cc:302] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c39374a36b24e2da8e814e842204ba3; no voters:
I20260928 02:39:00.775393 12189 leader_election.cc:288] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:00.775414 12194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:00.775602 12194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [term 1 LEADER]: Becoming Leader. State: Replica: 5c39374a36b24e2da8e814e842204ba3, State: Running, Role: LEADER
I20260928 02:39:00.775693 12189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:00.775708 12194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c39374a36b24e2da8e814e842204ba3" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38099 } }
I20260928 02:39:00.776060 12194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c39374a36b24e2da8e814e842204ba3. Latest consensus state: current_term: 1 leader_uuid: "5c39374a36b24e2da8e814e842204ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c39374a36b24e2da8e814e842204ba3" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38099 } } }
I20260928 02:39:00.776073 12195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c39374a36b24e2da8e814e842204ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c39374a36b24e2da8e814e842204ba3" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38099 } } }
I20260928 02:39:00.776171 12194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:00.776212 12195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:00.776412 12201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:00.776894 12201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:00.778170 12211 mvcc.cc:205] Tried to move back new op lower bound from 7334146624626077696 to 7334146624617648128. Current Snapshot: MvccSnapshot[applied={T|T < 7334146624626077696}]
I20260928 02:39:00.778586 12201 catalog_manager.cc:1347] Generated new cluster ID: ca2ed82cc37142d1ba04ff14f2087e68
I20260928 02:39:00.778635 12201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:39:00 dist-test-slave-kkrh krb5kdc[12108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:00.782430 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:00.771560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58574 (local address 127.9.130.62:38099)
0928 02:39:00.771738 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:00.771747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:00.771778 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:39:00.772321 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:00.772328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:00.772372 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:00.772531 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:00.773188 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:00.774596 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:00.775489 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:00.776376 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:00.776790 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:00.779267 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:00.779271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:00.779272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:00.779282 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:00.780947 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:00.781315 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:00.781317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:00.781318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:00.781406 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:00.781610 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:00.781611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:00.781611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:00.781833 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:00.781986 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:00.782105 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 02:39:00.782246 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260928 02:39:00.800449 12201 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:00.800974 12201 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:00.805776 12201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c39374a36b24e2da8e814e842204ba3: Generated new TSK 0
I20260928 02:39:00.805985 12201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:39:00.832653 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38099
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:00.929914 12215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:00.930112 12215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:00.930135 12215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:00.931268 12215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:00.931308 12215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:00.931334 12215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:00.932827 12215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:00.933074 12215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:00.933311 12215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:00.938323 12220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:00.938330 12221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:00.938494 12215 server_base.cc:1034] running on GCE node
W20260928 02:39:00.938331 12223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:00.938797 12215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:00.939050 12215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:00.940207 12215 hybrid_clock.cc:648] HybridClock initialized: now 1790563140940202 us; error 35 us; skew 500 ppm
Sep 28 02:39:00 dist-test-slave-kkrh krb5kdc[12108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563140, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:00.942693 12215 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:39:00.943177 12215 webserver.cc:460] Webserver started at http://127.9.130.1:36357/ using document root <none> and password file <none>
I20260928 02:39:00.943392 12215 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:00.943459 12215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:00.943607 12215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:00.944674 12215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "4a791a0a343648e0b20e4cb0a7fcc03c"
format_stamp: "Formatted at 2026-09-28 02:39:00 on dist-test-slave-kkrh"
server_key: "a37f43e873939623a8154587e9bced98"
server_key_iv: "a2e1d3b0a8ef30a5daa50fd61f401c78"
server_key_version: "encryptionkey@0"
I20260928 02:39:00.945034 12215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "4a791a0a343648e0b20e4cb0a7fcc03c"
format_stamp: "Formatted at 2026-09-28 02:39:00 on dist-test-slave-kkrh"
server_key: "a37f43e873939623a8154587e9bced98"
server_key_iv: "a2e1d3b0a8ef30a5daa50fd61f401c78"
server_key_version: "encryptionkey@0"
I20260928 02:39:00.946297 12215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:39:00.947031 12230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:00.947216 12215 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:00.947311 12215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "4a791a0a343648e0b20e4cb0a7fcc03c"
format_stamp: "Formatted at 2026-09-28 02:39:00 on dist-test-slave-kkrh"
server_key: "a37f43e873939623a8154587e9bced98"
server_key_iv: "a2e1d3b0a8ef30a5daa50fd61f401c78"
server_key_version: "encryptionkey@0"
I20260928 02:39:00.947395 12215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:00.971341 12215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:00.973336 12215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:00.973500 12215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:00.973737 12215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:00.974067 12215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:00.974124 12215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:00.974164 12215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:00.974191 12215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:00.980427 12215 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:35961
I20260928 02:39:00.980468 12343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:35961 every 8 connection(s)
I20260928 02:39:00.980773 12215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:39:00 dist-test-slave-kkrh krb5kdc[12108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563140, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:00.988281 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 12215
I20260928 02:39:00.988415 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:39:00.988626 9736 external_mini_cluster.cc:1428] Setting key 895569c259b9bc09823f6fadc396c7b2
I20260928 02:39:00.990263 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:00.981411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38099 (local address 127.9.130.1:48391)
0928 02:39:00.981586 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:00.981622 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:00.982109 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:00.982340 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:00.982344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:00.982890 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:00.983097 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:00.983101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:00.983803 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:00.983804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:00.984632 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:00.984634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:00.985019 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:00.985020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:00.985118 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:00.985575 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:00.985583 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:00.986930 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:00.988614 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:00.988619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:00.988621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:00.988895 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:00.989298 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:00.989299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:00.989300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:00.989419 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:00.989749 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:00.989755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:00.989878 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 02:39:00.989980 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:39:00.990105 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260928 02:39:00.990260 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:00.981420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:48391 (local address 127.9.130.62:38099)
0928 02:39:00.981486 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:00.981488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:00.981689 (+ 201us) server_negotiation.cc:407] Connection header received
0928 02:39:00.982197 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:00.982199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:00.982209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:00.982263 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:00.983204 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:00.983714 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:00.984714 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:00.984931 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:00.984989 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:00.987025 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:00.987026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:00.987026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:00.987033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:00.988524 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:00.989154 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:00.989155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:00.989155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:00.989216 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:00.989494 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:00.989495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:00.989496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:00.989643 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:00.989730 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:00.990023 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 02:39:00.990119 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:39:00.991207 12344 heartbeater.cc:347] Connected to a master server at 127.9.130.62:38099
I20260928 02:39:00.991290 12344 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:00.991477 12344 heartbeater.cc:511] Master 127.9.130.62:38099 requested a full tablet report, sending...
I20260928 02:39:00.992086 12153 ts_manager.cc:194] Registered new tserver with Master: 4a791a0a343648e0b20e4cb0a7fcc03c (127.9.130.1:35961)
I20260928 02:39:00.992707 12153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:48391
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:39:01.014279 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38099
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:01.093662 12349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:01.093863 12349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:01.093885 12349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:01.095019 12349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:01.095060 12349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:01.095086 12349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:01.096485 12349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:01.096719 12349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:01.096935 12349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:01.101687 12355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:01.101681 12357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:01.101725 12354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:01.101982 12349 server_base.cc:1034] running on GCE node
I20260928 02:39:01.102198 12349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:01.102415 12349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:01.103582 12349 hybrid_clock.cc:648] HybridClock initialized: now 1790563141103571 us; error 27 us; skew 500 ppm
Sep 28 02:39:01 dist-test-slave-kkrh krb5kdc[12108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563141, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:01.105993 12349 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:39:01.106467 12349 webserver.cc:460] Webserver started at http://127.9.130.2:43061/ using document root <none> and password file <none>
I20260928 02:39:01.106683 12349 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:01.106748 12349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:01.106902 12349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:01.108001 12349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "7c7ccb9f3f3a4158a25c3ac6f5064170"
format_stamp: "Formatted at 2026-09-28 02:39:01 on dist-test-slave-kkrh"
server_key: "3c03590168733508a1330c4965a174e3"
server_key_iv: "412e178fa98aa066226c75d8801767e6"
server_key_version: "encryptionkey@0"
I20260928 02:39:01.108368 12349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "7c7ccb9f3f3a4158a25c3ac6f5064170"
format_stamp: "Formatted at 2026-09-28 02:39:01 on dist-test-slave-kkrh"
server_key: "3c03590168733508a1330c4965a174e3"
server_key_iv: "412e178fa98aa066226c75d8801767e6"
server_key_version: "encryptionkey@0"
I20260928 02:39:01.109622 12349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:39:01.110352 12364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:01.110585 12349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:01.110684 12349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "7c7ccb9f3f3a4158a25c3ac6f5064170"
format_stamp: "Formatted at 2026-09-28 02:39:01 on dist-test-slave-kkrh"
server_key: "3c03590168733508a1330c4965a174e3"
server_key_iv: "412e178fa98aa066226c75d8801767e6"
server_key_version: "encryptionkey@0"
I20260928 02:39:01.110764 12349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:01.122557 12349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:01.124329 12349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:01.124482 12349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:01.124693 12349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:01.124977 12349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:01.125030 12349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:01.125072 12349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:01.125100 12349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:01.130976 12349 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:45049
I20260928 02:39:01.131027 12477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:45049 every 8 connection(s)
I20260928 02:39:01.131299 12349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:39:01 dist-test-slave-kkrh krb5kdc[12108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563141, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:01.139043 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 12349
I20260928 02:39:01.139156 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:39:01.139361 9736 external_mini_cluster.cc:1428] Setting key 1629732b42591f228b1926634f8b5ec9
I20260928 02:39:01.140028 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:01.131983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:59275 (local address 127.9.130.62:38099)
0928 02:39:01.132025 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:01.132027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:01.132198 (+ 171us) server_negotiation.cc:407] Connection header received
0928 02:39:01.132647 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:01.132648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:01.132656 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:01.132705 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:01.133568 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:01.134043 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:01.135014 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:01.135200 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:01.135254 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:01.137136 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:01.137137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:01.137138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:01.137144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:01.138593 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:01.138987 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:01.138987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:01.138988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:01.139034 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:01.139316 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:01.139317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:01.139317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:01.139419 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:01.139499 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:01.139853 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 02:39:01.139939 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:39:01.140236 12480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:01.131974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38099 (local address 127.9.130.2:59275)
0928 02:39:01.132107 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:01.132137 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:01.132563 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:01.132755 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:01.132758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:01.133284 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:01.133461 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:01.133465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:01.134132 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:01.134133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:01.134945 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:01.134948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:01.135271 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:01.135272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:01.135361 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:01.135746 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:01.135754 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:01.137059 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:01.138669 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:01.138674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:01.138675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:01.138910 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:01.139096 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:01.139097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:01.139097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:01.139242 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:01.139529 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:01.139534 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:01.139694 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 02:39:01.139808 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:39:01.139975 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260928 02:39:01.140717 12478 heartbeater.cc:347] Connected to a master server at 127.9.130.62:38099
I20260928 02:39:01.140798 12478 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:01.140977 12478 heartbeater.cc:511] Master 127.9.130.62:38099 requested a full tablet report, sending...
I20260928 02:39:01.141340 12153 ts_manager.cc:194] Registered new tserver with Master: 7c7ccb9f3f3a4158a25c3ac6f5064170 (127.9.130.2:45049)
I20260928 02:39:01.141722 12153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:59275
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:39:01.163332 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38099
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:01.244187 12483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:01.244411 12483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:01.244451 12483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:01.245679 12483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:01.245750 12483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:01.245796 12483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:01.247375 12483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:01.247663 12483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:01.247910 12483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:01.253154 12491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:01.253253 12483 server_base.cc:1034] running on GCE node
W20260928 02:39:01.253155 12488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:01.253216 12489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:01.253589 12483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:01.253826 12483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:01.254987 12483 hybrid_clock.cc:648] HybridClock initialized: now 1790563141254967 us; error 43 us; skew 500 ppm
Sep 28 02:39:01 dist-test-slave-kkrh krb5kdc[12108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563141, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:01.257539 12483 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:39:01.257980 12483 webserver.cc:460] Webserver started at http://127.9.130.3:40345/ using document root <none> and password file <none>
I20260928 02:39:01.258208 12483 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:01.258252 12483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:01.258340 12483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:01.259327 12483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "d9bf526036f647d8b3508d1c297070b7"
format_stamp: "Formatted at 2026-09-28 02:39:01 on dist-test-slave-kkrh"
server_key: "e1212edc6f0d37b82c06c13400ac6991"
server_key_iv: "2f36e16cd0035362afda8d8c8962aa08"
server_key_version: "encryptionkey@0"
I20260928 02:39:01.259617 12483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "d9bf526036f647d8b3508d1c297070b7"
format_stamp: "Formatted at 2026-09-28 02:39:01 on dist-test-slave-kkrh"
server_key: "e1212edc6f0d37b82c06c13400ac6991"
server_key_iv: "2f36e16cd0035362afda8d8c8962aa08"
server_key_version: "encryptionkey@0"
I20260928 02:39:01.260934 12483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:39:01.261672 12498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:01.261960 12483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:01.262050 12483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "d9bf526036f647d8b3508d1c297070b7"
format_stamp: "Formatted at 2026-09-28 02:39:01 on dist-test-slave-kkrh"
server_key: "e1212edc6f0d37b82c06c13400ac6991"
server_key_iv: "2f36e16cd0035362afda8d8c8962aa08"
server_key_version: "encryptionkey@0"
I20260928 02:39:01.262142 12483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:01.273578 12483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:01.275946 12483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:01.276096 12483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:01.276319 12483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:01.276784 12483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:01.276832 12483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:01.276863 12483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:01.276892 12483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:01.282971 12483 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:33899
I20260928 02:39:01.283010 12611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:33899 every 8 connection(s)
I20260928 02:39:01.283351 12483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:39:01.287778 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 12483
I20260928 02:39:01.287941 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:39:01.288125 9736 external_mini_cluster.cc:1428] Setting key cb0b04f645271d92062ceb1e2a8643bb
Sep 28 02:39:01 dist-test-slave-kkrh krb5kdc[12108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563141, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:01.293900 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:01.284841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:44325 (local address 127.9.130.62:38099)
0928 02:39:01.284908 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:01.284910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:01.285124 (+ 214us) server_negotiation.cc:407] Connection header received
0928 02:39:01.285699 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:01.285700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:01.285712 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:01.285756 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:01.286726 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:01.287251 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:01.288498 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:01.288857 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:01.288942 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:01.291002 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:01.291004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:01.291004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:01.291011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:01.292654 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:01.293002 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:01.293003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:01.293004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:01.293058 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:01.293290 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:01.293291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:01.293291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:01.293417 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:01.293490 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:01.293760 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 02:39:01.293834 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:39:01.294039 12614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:01.284811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38099 (local address 127.9.130.3:44325)
0928 02:39:01.284989 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:01.285035 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:01.285552 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:01.285828 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:01.285832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:01.286410 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:01.286607 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:01.286612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:01.287352 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:01.287353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:01.288404 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:01.288407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:01.288944 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:01.288945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:01.289110 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:01.289545 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:01.289552 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:01.290900 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:01.292727 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:01.292732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:01.292733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:01.292932 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:01.293117 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:01.293117 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:01.293118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:01.293208 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:01.293486 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:01.293489 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:01.293640 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 02:39:01.293721 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:39:01.293832 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260928 02:39:01.294543 12612 heartbeater.cc:347] Connected to a master server at 127.9.130.62:38099
I20260928 02:39:01.294624 12612 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:01.294806 12612 heartbeater.cc:511] Master 127.9.130.62:38099 requested a full tablet report, sending...
I20260928 02:39:01.295176 12153 ts_manager.cc:194] Registered new tserver with Master: d9bf526036f647d8b3508d1c297070b7 (127.9.130.3:33899)
I20260928 02:39:01.295493 12153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:44325
I20260928 02:39:01.299973 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:39:01.310979 12192 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:01.310149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.9.130.62:38099)
0928 02:39:01.310232 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:01.310234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:01.310257 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:39:01.310326 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:01.310327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:01.310336 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:01.310388 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:01.310914 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58576: BlockingRecv error: recv got EOF from 127.0.0.1:58576 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:39:01.311856 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 12215
I20260928 02:39:01.316226 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 12349
I20260928 02:39:01.320355 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 12483
I20260928 02:39:01.324488 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 12121
2026-09-28T02:39:01Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1826 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:39:01 dist-test-slave-kkrh krb5kdc[12624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:39:01 dist-test-slave-kkrh krb5kdc[12624](info): set up 2 sockets
Sep 28 02:39:01 dist-test-slave-kkrh krb5kdc[12624](info): commencing operation
krb5kdc: starting...
W20260928 02:39:02.368103 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:39:02 dist-test-slave-kkrh krb5kdc[12624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:39:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:39:02.480618 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:45071
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:45071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:02.558945 12637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:02.559129 12637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:02.559150 12637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:02.560235 12637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:02.560269 12637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:02.560283 12637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:02.560295 12637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:02.560307 12637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:02.561668 12637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:45071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:02.561832 12637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:02.561990 12637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:02.566779 12642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:02.566749 12645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:02.566954 12637 server_base.cc:1034] running on GCE node
W20260928 02:39:02.566843 12643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:02.567242 12637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:02.567560 12637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:02.568701 12637 hybrid_clock.cc:648] HybridClock initialized: now 1790563142568678 us; error 41 us; skew 500 ppm
Sep 28 02:39:02 dist-test-slave-kkrh krb5kdc[12624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563142, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:02.571111 12637 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:39:02.571560 12637 webserver.cc:460] Webserver started at http://127.9.130.62:41303/ using document root <none> and password file <none>
I20260928 02:39:02.571786 12637 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:02.571879 12637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:02.572023 12637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:02.573062 12637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "c72e39fc04db4fde87ff0e618988af13"
format_stamp: "Formatted at 2026-09-28 02:39:02 on dist-test-slave-kkrh"
server_key: "3c5de8f996004cfedf19d12314937a7e"
server_key_iv: "9514fb1807d0c12a5f33ae7c62382d06"
server_key_version: "encryptionkey@0"
I20260928 02:39:02.573413 12637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "c72e39fc04db4fde87ff0e618988af13"
format_stamp: "Formatted at 2026-09-28 02:39:02 on dist-test-slave-kkrh"
server_key: "3c5de8f996004cfedf19d12314937a7e"
server_key_iv: "9514fb1807d0c12a5f33ae7c62382d06"
server_key_version: "encryptionkey@0"
I20260928 02:39:02.574668 12637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:39:02.575484 12652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:02.575729 12637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:02.575809 12637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "c72e39fc04db4fde87ff0e618988af13"
format_stamp: "Formatted at 2026-09-28 02:39:02 on dist-test-slave-kkrh"
server_key: "3c5de8f996004cfedf19d12314937a7e"
server_key_iv: "9514fb1807d0c12a5f33ae7c62382d06"
server_key_version: "encryptionkey@0"
I20260928 02:39:02.575912 12637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:02.603416 12637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:02.605682 12637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:02.605796 12637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:02.609711 12637 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:45071
I20260928 02:39:02.609771 12704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:45071 every 8 connection(s)
I20260928 02:39:02.610110 12637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:02.610926 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:02.613166 12705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13: Bootstrap starting.
I20260928 02:39:02.613731 12705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:02.614001 12705 log.cc:824] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:02.614681 12705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13: No bootstrap required, opened a new log
I20260928 02:39:02.614698 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 12637
I20260928 02:39:02.614791 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:02.614972 9736 external_mini_cluster.cc:1428] Setting key 1677c2d3bc2a66d4f533fb093eb95054
I20260928 02:39:02.616933 12705 raft_consensus.cc:348] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72e39fc04db4fde87ff0e618988af13" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45071 } }
I20260928 02:39:02.617084 12705 raft_consensus.cc:374] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:02.617146 12705 raft_consensus.cc:729] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72e39fc04db4fde87ff0e618988af13, State: Initialized, Role: FOLLOWER
I20260928 02:39:02.617327 12705 consensus_queue.cc:260] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72e39fc04db4fde87ff0e618988af13" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45071 } }
I20260928 02:39:02.617419 12705 raft_consensus.cc:388] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:02.617465 12705 raft_consensus.cc:482] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:02.617511 12705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:02.618003 12705 raft_consensus.cc:504] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72e39fc04db4fde87ff0e618988af13" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45071 } }
I20260928 02:39:02.618114 12705 leader_election.cc:302] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c72e39fc04db4fde87ff0e618988af13; no voters:
I20260928 02:39:02.618276 12705 leader_election.cc:288] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:02.618330 12710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:02.618490 12710 raft_consensus.cc:686] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [term 1 LEADER]: Becoming Leader. State: Replica: c72e39fc04db4fde87ff0e618988af13, State: Running, Role: LEADER
I20260928 02:39:02.618526 12705 sys_catalog.cc:564] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:02.618638 12710 consensus_queue.cc:237] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72e39fc04db4fde87ff0e618988af13" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45071 } }
I20260928 02:39:02.619132 12712 sys_catalog.cc:455] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c72e39fc04db4fde87ff0e618988af13. Latest consensus state: current_term: 1 leader_uuid: "c72e39fc04db4fde87ff0e618988af13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72e39fc04db4fde87ff0e618988af13" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45071 } } }
I20260928 02:39:02.619230 12712 sys_catalog.cc:458] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:02.619457 12711 sys_catalog.cc:455] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c72e39fc04db4fde87ff0e618988af13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72e39fc04db4fde87ff0e618988af13" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45071 } } }
I20260928 02:39:02.619894 12719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:02.619926 12711 sys_catalog.cc:458] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:02.620409 12719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:39:02 dist-test-slave-kkrh krb5kdc[12624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:02.624878 12719 catalog_manager.cc:1347] Generated new cluster ID: 3fbd4089e29d4d5ea4acd8088974f404
I20260928 02:39:02.624938 12719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:39:02.626791 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:02.615468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.9.130.62:45071)
0928 02:39:02.615669 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:02.615676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:02.615704 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:39:02.616917 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:02.616924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:02.616971 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:02.617111 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:02.617799 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:02.619103 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:02.620209 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:02.620701 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:02.620828 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:02.623269 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:02.623272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:02.623272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:02.623282 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:02.625239 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:02.625607 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:02.625610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:02.625610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:02.625709 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:02.625934 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:02.625935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:02.625935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:02.626163 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:02.626331 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:02.626424 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 02:39:02.626648 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260928 02:39:02.627594 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:39:02.642037 12719 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:02.642776 12719 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:02.644632 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:02.640603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.9.130.62:45071)
0928 02:39:02.640642 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:02.640644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:02.640751 (+ 107us) server_negotiation.cc:407] Connection header received
0928 02:39:02.640800 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:02.640801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:02.640812 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:02.640883 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:02.641257 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:02.642011 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:02.643017 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:02.643332 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:02.643395 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:02.644146 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:02.644148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:02.644148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:02.644374 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:02.644391 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:02.644398 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:02.644454 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:02.644516 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 02:39:02.644555 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:39:02.652223 12719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c72e39fc04db4fde87ff0e618988af13: Generated new TSK 0
I20260928 02:39:02.652410 12719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:39:02.657128 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 12637
2026-09-28T02:39:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1330 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:39:02 dist-test-slave-kkrh krb5kdc[12738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:39:02 dist-test-slave-kkrh krb5kdc[12738](info): set up 2 sockets
Sep 28 02:39:02 dist-test-slave-kkrh krb5kdc[12738](info): commencing operation
krb5kdc: starting...
W20260928 02:39:03.803725 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:39:03 dist-test-slave-kkrh krb5kdc[12738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:39:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:39:03.918774 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:36651
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:36651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:03.996989 12751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:03.997238 12751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:03.997279 12751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:03.998427 12751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:03.998504 12751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:03.998531 12751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:03.998559 12751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:03.998595 12751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:04.000108 12751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:36651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:36651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:04.000393 12751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:04.000612 12751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:04.005492 12756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:04.005503 12759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:04.005502 12757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:04.005698 12751 server_base.cc:1034] running on GCE node
I20260928 02:39:04.006044 12751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:04.006412 12751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:04.007593 12751 hybrid_clock.cc:648] HybridClock initialized: now 1790563144007568 us; error 35 us; skew 500 ppm
Sep 28 02:39:04 dist-test-slave-kkrh krb5kdc[12738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563144, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:04.010334 12751 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:39:04.010802 12751 webserver.cc:460] Webserver started at http://127.9.130.62:43723/ using document root <none> and password file <none>
I20260928 02:39:04.011054 12751 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:04.011132 12751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:04.011296 12751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:04.012586 12751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "1d3f2a8e00114a958914578eb5497893"
format_stamp: "Formatted at 2026-09-28 02:39:04 on dist-test-slave-kkrh"
server_key: "cb0dae03cdfbb75f9743daa5a20cf46e"
server_key_iv: "10dbe3c009ca7d3869f6594e9b473aa2"
server_key_version: "encryptionkey@0"
I20260928 02:39:04.012979 12751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "1d3f2a8e00114a958914578eb5497893"
format_stamp: "Formatted at 2026-09-28 02:39:04 on dist-test-slave-kkrh"
server_key: "cb0dae03cdfbb75f9743daa5a20cf46e"
server_key_iv: "10dbe3c009ca7d3869f6594e9b473aa2"
server_key_version: "encryptionkey@0"
I20260928 02:39:04.014356 12751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:04.015072 12766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:04.015247 12751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:04.015367 12751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "1d3f2a8e00114a958914578eb5497893"
format_stamp: "Formatted at 2026-09-28 02:39:04 on dist-test-slave-kkrh"
server_key: "cb0dae03cdfbb75f9743daa5a20cf46e"
server_key_iv: "10dbe3c009ca7d3869f6594e9b473aa2"
server_key_version: "encryptionkey@0"
I20260928 02:39:04.015443 12751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:04.036886 12751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:04.038884 12751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:04.039049 12751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:04.042865 12751 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:36651
I20260928 02:39:04.042972 12818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:36651 every 8 connection(s)
I20260928 02:39:04.043268 12751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:04.044469 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:04.046697 12819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893: Bootstrap starting.
I20260928 02:39:04.047266 12819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:04.047547 12819 log.cc:824] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:04.048369 12819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893: No bootstrap required, opened a new log
I20260928 02:39:04.049891 12819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3f2a8e00114a958914578eb5497893" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36651 } }
I20260928 02:39:04.050072 12819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:04.050098 12819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3f2a8e00114a958914578eb5497893, State: Initialized, Role: FOLLOWER
I20260928 02:39:04.050246 12819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3f2a8e00114a958914578eb5497893" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36651 } }
I20260928 02:39:04.050308 12819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:04.050330 12819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:04.050374 12819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:04.051162 12819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3f2a8e00114a958914578eb5497893" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36651 } }
I20260928 02:39:04.051272 12819 leader_election.cc:302] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d3f2a8e00114a958914578eb5497893; no voters:
I20260928 02:39:04.051445 12819 leader_election.cc:288] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:04.051525 12822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:04.051683 12819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:04.051703 12822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [term 1 LEADER]: Becoming Leader. State: Replica: 1d3f2a8e00114a958914578eb5497893, State: Running, Role: LEADER
I20260928 02:39:04.051786 12822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3f2a8e00114a958914578eb5497893" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36651 } }
I20260928 02:39:04.052201 12823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d3f2a8e00114a958914578eb5497893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3f2a8e00114a958914578eb5497893" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36651 } } }
I20260928 02:39:04.052295 12823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:04.052495 12824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3f2a8e00114a958914578eb5497893. Latest consensus state: current_term: 1 leader_uuid: "1d3f2a8e00114a958914578eb5497893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3f2a8e00114a958914578eb5497893" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36651 } } }
I20260928 02:39:04.052525 12831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:04.052579 12824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:04.053042 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 12751
I20260928 02:39:04.053103 12831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:04.053151 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:04.053372 9736 external_mini_cluster.cc:1428] Setting key e1278429e7d19d75bd69f08f8826de44
I20260928 02:39:04.054811 12831 catalog_manager.cc:1347] Generated new cluster ID: b12198c115724037a14e667c8f9fc4fd
I20260928 02:39:04.054864 12831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:39:04 dist-test-slave-kkrh krb5kdc[12738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:04.065467 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:04.053828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.9.130.62:36651)
0928 02:39:04.054230 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:04.054239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:04.054268 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:39:04.054430 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:04.054439 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:04.054514 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:04.054663 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:04.055470 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.056989 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.058073 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.058743 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.058853 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:04.061800 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:04.061804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:04.061804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:04.061816 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:04.063870 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:04.064244 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:04.064246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:04.064247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:04.064352 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:04.064544 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:04.064545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:04.064546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:04.064842 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:04.064992 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:04.065094 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:39:04.065229 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":73,"threads_started":1}
I20260928 02:39:04.072901 12831 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:04.073423 12831 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:04.080224 12831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d3f2a8e00114a958914578eb5497893: Generated new TSK 0
I20260928 02:39:04.080403 12831 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:39:04.111040 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:36651
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:04.208933 12845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:04.209116 12845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:04.209138 12845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:04.210227 12845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:04.210260 12845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:04.210285 12845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:04.211618 12845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:36651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:04.211787 12845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:04.212025 12845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:04.216691 12853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:04.216742 12851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:04.216885 12850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:04.216849 12845 server_base.cc:1034] running on GCE node
I20260928 02:39:04.217154 12845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:04.217386 12845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:04.218535 12845 hybrid_clock.cc:648] HybridClock initialized: now 1790563144218523 us; error 37 us; skew 500 ppm
Sep 28 02:39:04 dist-test-slave-kkrh krb5kdc[12738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563144, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:04.221014 12845 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:39:04.221437 12845 webserver.cc:460] Webserver started at http://127.9.130.1:39503/ using document root <none> and password file <none>
I20260928 02:39:04.221623 12845 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:04.221662 12845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:04.221746 12845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:04.222692 12845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "c842849b857e4facbb02fd52fe810087"
format_stamp: "Formatted at 2026-09-28 02:39:04 on dist-test-slave-kkrh"
server_key: "cae877dbdbf1c5e94c3640d3d1cc23bb"
server_key_iv: "d1b0b0bf28c76a313df59baec789e458"
server_key_version: "encryptionkey@0"
I20260928 02:39:04.222977 12845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "c842849b857e4facbb02fd52fe810087"
format_stamp: "Formatted at 2026-09-28 02:39:04 on dist-test-slave-kkrh"
server_key: "cae877dbdbf1c5e94c3640d3d1cc23bb"
server_key_iv: "d1b0b0bf28c76a313df59baec789e458"
server_key_version: "encryptionkey@0"
I20260928 02:39:04.224220 12845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:39:04.224905 12860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:04.225113 12845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:04.225225 12845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "c842849b857e4facbb02fd52fe810087"
format_stamp: "Formatted at 2026-09-28 02:39:04 on dist-test-slave-kkrh"
server_key: "cae877dbdbf1c5e94c3640d3d1cc23bb"
server_key_iv: "d1b0b0bf28c76a313df59baec789e458"
server_key_version: "encryptionkey@0"
I20260928 02:39:04.225315 12845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:04.239368 12845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:04.241557 12845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:04.241707 12845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:04.241938 12845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:04.242223 12845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:04.242270 12845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:04.242309 12845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:04.242338 12845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:04.248262 12845 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:40477
I20260928 02:39:04.248306 12973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:40477 every 8 connection(s)
I20260928 02:39:04.248603 12845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:39:04 dist-test-slave-kkrh krb5kdc[12738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563144, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:04.256049 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 12845
I20260928 02:39:04.256130 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:39:04.256351 9736 external_mini_cluster.cc:1428] Setting key e0c25df1f1dbefc3661c6af9fbe60991
I20260928 02:39:04.258186 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:04.249287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:42423 (local address 127.9.130.62:36651)
0928 02:39:04.249382 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:04.249385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:04.249574 (+ 189us) server_negotiation.cc:407] Connection header received
0928 02:39:04.250066 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:04.250068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:04.250082 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:04.250139 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:04.251010 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.251516 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.252790 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.253033 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.253100 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:04.255136 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:04.255137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:04.255137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:04.255145 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:04.256613 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:04.257084 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:04.257085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:04.257085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:04.257140 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:04.257404 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:04.257405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:04.257405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:04.257666 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:04.257738 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:04.258061 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 02:39:04.258126 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:39:04.258469 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:04.249270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:36651 (local address 127.9.130.1:42423)
0928 02:39:04.249452 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:04.249485 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:04.249956 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:04.250191 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:04.250194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:04.250742 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:04.250931 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:04.250935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.251651 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.251652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:04.252665 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:04.252667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.253123 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.253123 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:04.253253 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:04.253761 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:04.253770 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:04.255059 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:04.256691 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:04.256698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:04.256701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:04.257003 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:04.257201 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:04.257202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:04.257202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:04.257341 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:04.257746 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:04.257751 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:04.257919 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 02:39:04.258033 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:39:04.258201 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260928 02:39:04.259096 12974 heartbeater.cc:347] Connected to a master server at 127.9.130.62:36651
I20260928 02:39:04.259171 12974 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:04.259337 12974 heartbeater.cc:511] Master 127.9.130.62:36651 requested a full tablet report, sending...
I20260928 02:39:04.259891 12783 ts_manager.cc:194] Registered new tserver with Master: c842849b857e4facbb02fd52fe810087 (127.9.130.1:40477)
I20260928 02:39:04.260828 12783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:42423
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:39:04.282248 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:36651
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:04.360845 12979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:04.361037 12979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:04.361059 12979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:04.362205 12979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:04.362246 12979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:04.362272 12979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:04.363662 12979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:36651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:04.363909 12979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:04.364137 12979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:04.368971 12987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:04.369065 12985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:04.368971 12984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:04.369333 12979 server_base.cc:1034] running on GCE node
I20260928 02:39:04.369550 12979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:04.369764 12979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:04.370955 12979 hybrid_clock.cc:648] HybridClock initialized: now 1790563144370928 us; error 41 us; skew 500 ppm
Sep 28 02:39:04 dist-test-slave-kkrh krb5kdc[12738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563144, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:04.373617 12979 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:39:04.374025 12979 webserver.cc:460] Webserver started at http://127.9.130.2:41475/ using document root <none> and password file <none>
I20260928 02:39:04.374249 12979 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:04.374312 12979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:04.374447 12979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:04.375447 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "e5c15e44158040f6a37977fa83565f7d"
format_stamp: "Formatted at 2026-09-28 02:39:04 on dist-test-slave-kkrh"
server_key: "2dfe5ce023c2f23e67caed030dab1709"
server_key_iv: "f8647a84932a1b523e762530b2e00371"
server_key_version: "encryptionkey@0"
I20260928 02:39:04.375801 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "e5c15e44158040f6a37977fa83565f7d"
format_stamp: "Formatted at 2026-09-28 02:39:04 on dist-test-slave-kkrh"
server_key: "2dfe5ce023c2f23e67caed030dab1709"
server_key_iv: "f8647a84932a1b523e762530b2e00371"
server_key_version: "encryptionkey@0"
I20260928 02:39:04.377089 12979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:04.377767 12994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:04.377923 12979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:04.378012 12979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "e5c15e44158040f6a37977fa83565f7d"
format_stamp: "Formatted at 2026-09-28 02:39:04 on dist-test-slave-kkrh"
server_key: "2dfe5ce023c2f23e67caed030dab1709"
server_key_iv: "f8647a84932a1b523e762530b2e00371"
server_key_version: "encryptionkey@0"
I20260928 02:39:04.378083 12979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:04.387566 12979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:04.389293 12979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:04.389441 12979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:04.389647 12979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:04.389951 12979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:04.390004 12979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:04.390045 12979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:04.390070 12979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:04.395745 12979 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:38205
I20260928 02:39:04.395784 13107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:38205 every 8 connection(s)
I20260928 02:39:04.396100 12979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:39:04.396473 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 12979
I20260928 02:39:04.396572 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:39:04.396792 9736 external_mini_cluster.cc:1428] Setting key 07d476ca09e8d8144de0c72927813d23
Sep 28 02:39:04 dist-test-slave-kkrh krb5kdc[12738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563144, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
I20260928 02:39:04.405836 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:04.397052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:59237 (local address 127.9.130.62:36651)
0928 02:39:04.397106 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:04.397108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:04.397179 (+ 71us) server_negotiation.cc:407] Connection header received
0928 02:39:04.397795 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:04.397796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:04.397806 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:04.397860 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:04.398762 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.399261 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.400330 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.400533 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.400613 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:04.402900 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:04.402902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:04.402902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:04.402910 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:04.404517 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:04.404921 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:04.404922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:04.404922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:04.404989 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:04.405264 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:04.405265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:04.405265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:04.405419 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:04.405477 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:04.405704 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 02:39:04.405772 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:39:04.406000 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:04.396889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:36651 (local address 127.9.130.2:59237)
0928 02:39:04.397065 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:04.397114 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:04.397694 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:04.397928 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:04.397932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:04.398474 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:04.398667 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:04.398671 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.399350 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.399351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:04.400236 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:04.400238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.400620 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.400621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:04.400742 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:04.401220 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:04.401227 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:04.402809 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:04.404627 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:04.404632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:04.404634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:04.404849 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:04.405081 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:04.405082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:04.405083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:04.405196 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:04.405481 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:04.405485 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:04.405593 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 02:39:04.405675 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:39:04.405787 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260928 02:39:04.406478 13108 heartbeater.cc:347] Connected to a master server at 127.9.130.62:36651
I20260928 02:39:04.406572 13108 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:04.406750 13108 heartbeater.cc:511] Master 127.9.130.62:36651 requested a full tablet report, sending...
I20260928 02:39:04.407208 12783 ts_manager.cc:194] Registered new tserver with Master: e5c15e44158040f6a37977fa83565f7d (127.9.130.2:38205)
I20260928 02:39:04.407577 12783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:59237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:39:04.420758 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:36651
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:04.502389 13113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:04.502579 13113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:04.502602 13113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:04.503690 13113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:04.503723 13113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:04.503748 13113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:04.505146 13113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:36651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:04.505322 13113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:04.505491 13113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:04.510269 13118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:04.510259 13121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:04.510376 13119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:04.510429 13113 server_base.cc:1034] running on GCE node
I20260928 02:39:04.510673 13113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:04.510905 13113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:04.512089 13113 hybrid_clock.cc:648] HybridClock initialized: now 1790563144512073 us; error 27 us; skew 500 ppm
Sep 28 02:39:04 dist-test-slave-kkrh krb5kdc[12738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563144, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:04.514436 13113 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:39:04.514895 13113 webserver.cc:460] Webserver started at http://127.9.130.3:36473/ using document root <none> and password file <none>
I20260928 02:39:04.515084 13113 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:04.515123 13113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:04.515213 13113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:04.516244 13113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "02ee3aef007e420c9b84752283205074"
format_stamp: "Formatted at 2026-09-28 02:39:04 on dist-test-slave-kkrh"
server_key: "b54ee945f78fbaca528a03205e11b749"
server_key_iv: "50d278cafb7750c86419e7b5cab20059"
server_key_version: "encryptionkey@0"
I20260928 02:39:04.516541 13113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "02ee3aef007e420c9b84752283205074"
format_stamp: "Formatted at 2026-09-28 02:39:04 on dist-test-slave-kkrh"
server_key: "b54ee945f78fbaca528a03205e11b749"
server_key_iv: "50d278cafb7750c86419e7b5cab20059"
server_key_version: "encryptionkey@0"
I20260928 02:39:04.517755 13113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:04.518438 13128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:04.518622 13113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:04.518699 13113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "02ee3aef007e420c9b84752283205074"
format_stamp: "Formatted at 2026-09-28 02:39:04 on dist-test-slave-kkrh"
server_key: "b54ee945f78fbaca528a03205e11b749"
server_key_iv: "50d278cafb7750c86419e7b5cab20059"
server_key_version: "encryptionkey@0"
I20260928 02:39:04.518754 13113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:04.542091 13113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:04.544430 13113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:04.544545 13113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:04.544736 13113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:04.545083 13113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:04.545135 13113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:04.545190 13113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:04.545218 13113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:04.550817 13113 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:33699
I20260928 02:39:04.550912 13241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:33699 every 8 connection(s)
I20260928 02:39:04.551203 13113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:39:04.555382 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 13113
I20260928 02:39:04.555483 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:39:04.555704 9736 external_mini_cluster.cc:1428] Setting key 9f64c36fdda590e078a0290a743b9d63
Sep 28 02:39:04 dist-test-slave-kkrh krb5kdc[12738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563144, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:04.560395 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:04.551921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:38247 (local address 127.9.130.62:36651)
0928 02:39:04.551973 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:04.551975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:04.552212 (+ 237us) server_negotiation.cc:407] Connection header received
0928 02:39:04.552801 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:04.552802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:04.552811 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:04.552864 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:04.553799 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.554279 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.555250 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.555560 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.555647 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:04.557866 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:04.557867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:04.557868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:04.557875 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:04.559025 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:04.559424 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:04.559424 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:04.559425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:04.559484 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:04.559723 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:04.559724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:04.559724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:04.559889 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:04.559953 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:04.560225 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 02:39:04.560303 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:39:04.560591 13244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:04.551938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:36651 (local address 127.9.130.3:38247)
0928 02:39:04.552098 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:04.552133 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:04.552712 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:04.552967 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:04.552971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:04.553523 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:04.553720 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:04.553725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.554353 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.554353 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:04.555149 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:04.555151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.555649 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.555650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:04.555777 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:04.556396 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:04.556406 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:04.557759 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:04.559118 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:04.559124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:04.559126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:04.559342 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:04.559548 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:04.559549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:04.559549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:04.559657 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:04.559950 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:04.559957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:04.560098 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 02:39:04.560204 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:39:04.560358 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260928 02:39:04.561010 13242 heartbeater.cc:347] Connected to a master server at 127.9.130.62:36651
I20260928 02:39:04.561102 13242 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:04.561234 13242 heartbeater.cc:511] Master 127.9.130.62:36651 requested a full tablet report, sending...
I20260928 02:39:04.561580 12783 ts_manager.cc:194] Registered new tserver with Master: 02ee3aef007e420c9b84752283205074 (127.9.130.3:33699)
I20260928 02:39:04.561933 12783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:38247
I20260928 02:39:04.567433 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:39:04 dist-test-slave-kkrh krb5kdc[12738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563144, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 02:39:04 dist-test-slave-kkrh krb5kdc[12738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563144, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:04.589035 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:04.582431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.9.130.62:36651)
0928 02:39:04.582503 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:04.582505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:04.582611 (+ 106us) server_negotiation.cc:407] Connection header received
0928 02:39:04.582703 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:04.582704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:04.582715 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:04.582780 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:04.583365 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.583909 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.584654 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:04.584849 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:04.584920 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:04.586947 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:04.586948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:04.586949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:04.586954 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:04.588139 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:04.588433 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:04.588434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:04.588435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:04.588479 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:04.588644 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:04.588645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:04.588645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:04.588767 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:04.588838 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:04.588916 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 02:39:04.588971 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 02:39:04.589264 12783 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37302
I20260928 02:39:04.589696 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 12845
I20260928 02:39:04.594138 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 12979
I20260928 02:39:04.598227 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 13113
I20260928 02:39:04.601923 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 12751
2026-09-28T02:39:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1946 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:39:04 dist-test-slave-kkrh krb5kdc[13254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:39:04 dist-test-slave-kkrh krb5kdc[13254](info): set up 2 sockets
Sep 28 02:39:04 dist-test-slave-kkrh krb5kdc[13254](info): commencing operation
krb5kdc: starting...
W20260928 02:39:05.632121 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:39:05 dist-test-slave-kkrh krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:39:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:39:05.748540 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:34521
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:34521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:05.845992 13267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:05.846222 13267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:05.846256 13267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:05.847384 13267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:05.847448 13267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:05.847471 13267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:05.847491 13267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:05.847520 13267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:05.849018 13267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:34521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:34521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:05.849280 13267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:05.849540 13267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:05.854429 13273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:05.854436 13272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:05.854504 13275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:05.854702 13267 server_base.cc:1034] running on GCE node
I20260928 02:39:05.854969 13267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:05.855255 13267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:05.856403 13267 hybrid_clock.cc:648] HybridClock initialized: now 1790563145856385 us; error 39 us; skew 500 ppm
Sep 28 02:39:05 dist-test-slave-kkrh krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563145, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:05.858759 13267 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:39:05.859177 13267 webserver.cc:460] Webserver started at http://127.9.130.62:38531/ using document root <none> and password file <none>
I20260928 02:39:05.859407 13267 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:05.859469 13267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:05.859609 13267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:05.860709 13267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "438166fa44584ccc8512914d820f57f5"
format_stamp: "Formatted at 2026-09-28 02:39:05 on dist-test-slave-kkrh"
server_key: "cd80b15dc43fffe8d17f279273f1d128"
server_key_iv: "d8563aec6a7e7dac65bb23a1ce7b3939"
server_key_version: "encryptionkey@0"
I20260928 02:39:05.861066 13267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "438166fa44584ccc8512914d820f57f5"
format_stamp: "Formatted at 2026-09-28 02:39:05 on dist-test-slave-kkrh"
server_key: "cd80b15dc43fffe8d17f279273f1d128"
server_key_iv: "d8563aec6a7e7dac65bb23a1ce7b3939"
server_key_version: "encryptionkey@0"
I20260928 02:39:05.862354 13267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:05.863046 13282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:05.863204 13267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:05.863296 13267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "438166fa44584ccc8512914d820f57f5"
format_stamp: "Formatted at 2026-09-28 02:39:05 on dist-test-slave-kkrh"
server_key: "cd80b15dc43fffe8d17f279273f1d128"
server_key_iv: "d8563aec6a7e7dac65bb23a1ce7b3939"
server_key_version: "encryptionkey@0"
I20260928 02:39:05.863372 13267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:05.878229 13267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:05.880151 13267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:05.880316 13267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:05.884065 13267 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:34521
I20260928 02:39:05.884115 13334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:34521 every 8 connection(s)
I20260928 02:39:05.884629 13267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:05.885239 13335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:05.887410 13335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5: Bootstrap starting.
I20260928 02:39:05.888033 13335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:05.888311 13335 log.cc:824] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:05.888954 13335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5: No bootstrap required, opened a new log
I20260928 02:39:05.890476 13335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438166fa44584ccc8512914d820f57f5" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34521 } }
I20260928 02:39:05.890628 13335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:05.890683 13335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 438166fa44584ccc8512914d820f57f5, State: Initialized, Role: FOLLOWER
I20260928 02:39:05.890887 13335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438166fa44584ccc8512914d820f57f5" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34521 } }
I20260928 02:39:05.890985 13335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:05.891021 13335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:05.891078 13335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:05.892316 13335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438166fa44584ccc8512914d820f57f5" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34521 } }
I20260928 02:39:05.892408 13335 leader_election.cc:302] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 438166fa44584ccc8512914d820f57f5; no voters:
I20260928 02:39:05.892534 13335 leader_election.cc:288] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:05.892599 13338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:05.892793 13335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:05.892792 13338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [term 1 LEADER]: Becoming Leader. State: Replica: 438166fa44584ccc8512914d820f57f5, State: Running, Role: LEADER
I20260928 02:39:05.892987 13338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438166fa44584ccc8512914d820f57f5" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34521 } }
I20260928 02:39:05.893110 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 13267
I20260928 02:39:05.893230 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:05.893441 9736 external_mini_cluster.cc:1428] Setting key e7aa9b77ee15d5c2fb550db859dbfb02
I20260928 02:39:05.893407 13340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 438166fa44584ccc8512914d820f57f5. Latest consensus state: current_term: 1 leader_uuid: "438166fa44584ccc8512914d820f57f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438166fa44584ccc8512914d820f57f5" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34521 } } }
I20260928 02:39:05.893481 13340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:05.894229 13339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "438166fa44584ccc8512914d820f57f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438166fa44584ccc8512914d820f57f5" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34521 } } }
I20260928 02:39:05.894353 13339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5 [sys.catalog]: This master's current role is: LEADER
W20260928 02:39:05.894966 13356 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:39:05.895032 13356 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:39:05.895169 13346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:05.895350 13346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:05.896891 13346 catalog_manager.cc:1347] Generated new cluster ID: 7954f402d96a4e6f82ce93c0285e3fae
I20260928 02:39:05.896955 13346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:39:05 dist-test-slave-kkrh krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:05.903420 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:05.893978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.9.130.62:34521)
0928 02:39:05.894180 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:05.894190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:05.894229 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:39:05.894351 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:05.894359 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:05.894420 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:05.894561 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:05.895357 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:05.896545 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:05.897464 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:05.897894 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:05.898003 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:05.900244 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:05.900248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:05.900248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:05.900260 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:05.901928 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:05.902288 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:05.902291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:05.902292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:05.902376 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:05.902575 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:05.902576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:05.902576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:05.902800 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:05.902967 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:05.903030 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 02:39:05.903268 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":59,"threads_started":1}
I20260928 02:39:05.915854 13346 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:05.916548 13346 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:05.925069 13346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 438166fa44584ccc8512914d820f57f5: Generated new TSK 0
I20260928 02:39:05.925240 13346 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:39:05.954900 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34521
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:06.033573 13361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:06.033798 13361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:06.033833 13361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:06.035005 13361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:06.035063 13361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:06.035106 13361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:06.036602 13361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:06.036854 13361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:06.037074 13361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:06.041953 13369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:06.041939 13366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:06.041941 13367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:06.042312 13361 server_base.cc:1034] running on GCE node
I20260928 02:39:06.042524 13361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:06.042724 13361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:06.043876 13361 hybrid_clock.cc:648] HybridClock initialized: now 1790563146043863 us; error 30 us; skew 500 ppm
Sep 28 02:39:06 dist-test-slave-kkrh krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563146, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:06.046270 13361 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:39:06.046703 13361 webserver.cc:460] Webserver started at http://127.9.130.1:37693/ using document root <none> and password file <none>
I20260928 02:39:06.046881 13361 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:06.046921 13361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:06.047008 13361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:06.048058 13361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "f3f779f5a9004791aad79cfe1cf6f3e5"
format_stamp: "Formatted at 2026-09-28 02:39:06 on dist-test-slave-kkrh"
server_key: "de7ce5941360f015ca4a497548cf6a54"
server_key_iv: "a0c827f902c1710a934699a3cc1015d2"
server_key_version: "encryptionkey@0"
I20260928 02:39:06.048367 13361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "f3f779f5a9004791aad79cfe1cf6f3e5"
format_stamp: "Formatted at 2026-09-28 02:39:06 on dist-test-slave-kkrh"
server_key: "de7ce5941360f015ca4a497548cf6a54"
server_key_iv: "a0c827f902c1710a934699a3cc1015d2"
server_key_version: "encryptionkey@0"
I20260928 02:39:06.049582 13361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:06.050204 13376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:06.050365 13361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:06.050443 13361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "f3f779f5a9004791aad79cfe1cf6f3e5"
format_stamp: "Formatted at 2026-09-28 02:39:06 on dist-test-slave-kkrh"
server_key: "de7ce5941360f015ca4a497548cf6a54"
server_key_iv: "a0c827f902c1710a934699a3cc1015d2"
server_key_version: "encryptionkey@0"
I20260928 02:39:06.050498 13361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:06.077049 13361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:06.078962 13361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:06.079077 13361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:06.079306 13361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:06.079636 13361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:06.079671 13361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:06.079696 13361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:06.079710 13361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:06.085497 13361 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:45141
I20260928 02:39:06.085543 13489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:45141 every 8 connection(s)
I20260928 02:39:06.085840 13361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:39:06.089596 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 13361
I20260928 02:39:06.089687 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:39:06.089876 9736 external_mini_cluster.cc:1428] Setting key f456cfbe394ada3fe060635f62e5407e
Sep 28 02:39:06 dist-test-slave-kkrh krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563146, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:06.096210 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:06.086471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:60797 (local address 127.9.130.62:34521)
0928 02:39:06.086531 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:06.086533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:06.086717 (+ 184us) server_negotiation.cc:407] Connection header received
0928 02:39:06.087657 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:06.087659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:06.087670 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:06.087713 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:06.088338 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.088851 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.089905 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.090241 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.090313 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:06.093140 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:06.093147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:06.093148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:06.093155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:06.094795 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:06.095202 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:06.095203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:06.095204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:06.095266 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:06.095506 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:06.095507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:06.095507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:06.095629 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:06.095700 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:06.096020 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 02:39:06.096104 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:39:06.096405 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:06.086462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34521 (local address 127.9.130.1:60797)
0928 02:39:06.086596 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:06.086634 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:06.087552 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:06.087793 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:06.087797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:06.088058 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:06.088244 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:06.088248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.088949 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.088950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:06.089822 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:06.089825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.090320 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.090321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:06.090463 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:06.090974 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:06.090984 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:06.093024 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:06.094885 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:06.094892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:06.094894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:06.095125 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:06.095329 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:06.095330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:06.095331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:06.095442 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:06.095693 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:06.095699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:06.095883 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 02:39:06.095999 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:39:06.096161 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
I20260928 02:39:06.097049 13490 heartbeater.cc:347] Connected to a master server at 127.9.130.62:34521
I20260928 02:39:06.097142 13490 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:06.097337 13490 heartbeater.cc:511] Master 127.9.130.62:34521 requested a full tablet report, sending...
I20260928 02:39:06.097877 13299 ts_manager.cc:194] Registered new tserver with Master: f3f779f5a9004791aad79cfe1cf6f3e5 (127.9.130.1:45141)
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
I20260928 02:39:06.098686 13299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:60797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:39:06.114441 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34521
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:06.193498 13495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:06.193686 13495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:06.193708 13495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:06.194937 13495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:06.194980 13495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:06.195006 13495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:06.196456 13495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:06.196627 13495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:06.196803 13495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:06.202268 13500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:06.202251 13501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:06.202268 13503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:06.202695 13495 server_base.cc:1034] running on GCE node
I20260928 02:39:06.202921 13495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:06.203140 13495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:06.204288 13495 hybrid_clock.cc:648] HybridClock initialized: now 1790563146204269 us; error 36 us; skew 500 ppm
Sep 28 02:39:06 dist-test-slave-kkrh krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563146, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:06.206488 13495 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:39:06.206919 13495 webserver.cc:460] Webserver started at http://127.9.130.2:45389/ using document root <none> and password file <none>
I20260928 02:39:06.207129 13495 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:06.207187 13495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:06.207324 13495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:06.208345 13495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "5c7fdf7303bb4ee8b7595bb397100e12"
format_stamp: "Formatted at 2026-09-28 02:39:06 on dist-test-slave-kkrh"
server_key: "97c2ecbe995a8760e762b9f033b6c027"
server_key_iv: "1341bee69cfd1e86635a8a46aa011f87"
server_key_version: "encryptionkey@0"
I20260928 02:39:06.208683 13495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "5c7fdf7303bb4ee8b7595bb397100e12"
format_stamp: "Formatted at 2026-09-28 02:39:06 on dist-test-slave-kkrh"
server_key: "97c2ecbe995a8760e762b9f033b6c027"
server_key_iv: "1341bee69cfd1e86635a8a46aa011f87"
server_key_version: "encryptionkey@0"
I20260928 02:39:06.209908 13495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:06.210843 13510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:06.211017 13495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:06.211118 13495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "5c7fdf7303bb4ee8b7595bb397100e12"
format_stamp: "Formatted at 2026-09-28 02:39:06 on dist-test-slave-kkrh"
server_key: "97c2ecbe995a8760e762b9f033b6c027"
server_key_iv: "1341bee69cfd1e86635a8a46aa011f87"
server_key_version: "encryptionkey@0"
I20260928 02:39:06.211195 13495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:06.226006 13495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:06.227702 13495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:06.227882 13495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:06.228089 13495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:06.228376 13495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:06.228432 13495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:06.228471 13495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:06.228502 13495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:06.234201 13495 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:36323
I20260928 02:39:06.234237 13623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:36323 every 8 connection(s)
I20260928 02:39:06.234543 13495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:39:06.238644 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 13495
I20260928 02:39:06.238761 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:39:06.238950 9736 external_mini_cluster.cc:1428] Setting key bde8c694b370ad4acd4893da199cea0d
Sep 28 02:39:06 dist-test-slave-kkrh krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563146, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:06.244717 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:06.235202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34521 (local address 127.9.130.2:40715)
0928 02:39:06.235394 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:06.235437 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:06.236372 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:06.236629 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:06.236633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:06.236879 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:06.237052 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:06.237055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.237751 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.237752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:06.238682 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:06.238685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.239124 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.239124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:06.239284 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:06.239810 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:06.239852 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:06.241615 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:06.243422 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:06.243427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:06.243429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:06.243634 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:06.243881 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:06.243882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:06.243882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:06.243976 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:06.244265 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:06.244269 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:06.244374 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 02:39:06.244450 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:39:06.244564 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260928 02:39:06.244738 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:06.235294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:40715 (local address 127.9.130.62:34521)
0928 02:39:06.235364 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:06.235366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:06.235528 (+ 162us) server_negotiation.cc:407] Connection header received
0928 02:39:06.236483 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:06.236485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:06.236497 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:06.236561 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:06.237145 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.237637 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.238786 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.239048 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.239106 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:06.241716 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:06.241718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:06.241718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:06.241724 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:06.243337 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:06.243712 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:06.243714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:06.243714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:06.243776 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:06.244040 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:06.244041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:06.244042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:06.244191 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:06.244263 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:06.244496 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 02:39:06.244588 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:39:06.245405 13624 heartbeater.cc:347] Connected to a master server at 127.9.130.62:34521
I20260928 02:39:06.245509 13624 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:06.245671 13624 heartbeater.cc:511] Master 127.9.130.62:34521 requested a full tablet report, sending...
I20260928 02:39:06.246047 13299 ts_manager.cc:194] Registered new tserver with Master: 5c7fdf7303bb4ee8b7595bb397100e12 (127.9.130.2:36323)
I20260928 02:39:06.246410 13299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:40715
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:39:06.264753 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34521
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:06.344877 13629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:06.345108 13629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:06.345145 13629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:06.346283 13629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:06.346346 13629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:06.346390 13629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:06.347882 13629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:06.348143 13629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:06.348364 13629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:06.353164 13634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:06.353169 13635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:06.353214 13637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:06.353484 13629 server_base.cc:1034] running on GCE node
I20260928 02:39:06.353731 13629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:06.353971 13629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:06.355147 13629 hybrid_clock.cc:648] HybridClock initialized: now 1790563146355133 us; error 28 us; skew 500 ppm
Sep 28 02:39:06 dist-test-slave-kkrh krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563146, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:06.357899 13629 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:39:06.358335 13629 webserver.cc:460] Webserver started at http://127.9.130.3:41277/ using document root <none> and password file <none>
I20260928 02:39:06.358599 13629 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:06.358664 13629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:06.358834 13629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:06.359871 13629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "a3301a26651a4a738c62cae9d01df586"
format_stamp: "Formatted at 2026-09-28 02:39:06 on dist-test-slave-kkrh"
server_key: "7d5007877bef7806be8a77b9fadb77bc"
server_key_iv: "89a478534b5b23e4f83f0ff01e423a65"
server_key_version: "encryptionkey@0"
I20260928 02:39:06.360237 13629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "a3301a26651a4a738c62cae9d01df586"
format_stamp: "Formatted at 2026-09-28 02:39:06 on dist-test-slave-kkrh"
server_key: "7d5007877bef7806be8a77b9fadb77bc"
server_key_iv: "89a478534b5b23e4f83f0ff01e423a65"
server_key_version: "encryptionkey@0"
I20260928 02:39:06.361509 13629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:06.362257 13644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:06.362679 13629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:06.362790 13629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "a3301a26651a4a738c62cae9d01df586"
format_stamp: "Formatted at 2026-09-28 02:39:06 on dist-test-slave-kkrh"
server_key: "7d5007877bef7806be8a77b9fadb77bc"
server_key_iv: "89a478534b5b23e4f83f0ff01e423a65"
server_key_version: "encryptionkey@0"
I20260928 02:39:06.362862 13629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:06.385787 13629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:06.387866 13629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:06.388025 13629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:06.388255 13629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:06.388552 13629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:06.388603 13629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:06.388645 13629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:06.388671 13629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:06.394503 13629 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:38813
I20260928 02:39:06.394567 13757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:38813 every 8 connection(s)
I20260928 02:39:06.394857 13629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:39:06.399433 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 13629
I20260928 02:39:06.399546 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:39:06.399739 9736 external_mini_cluster.cc:1428] Setting key 577a2dad51c5522c94a05d93d0f15d96
Sep 28 02:39:06 dist-test-slave-kkrh krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563146, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:06.404237 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:06.395497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:49341 (local address 127.9.130.62:34521)
0928 02:39:06.395555 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:06.395557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:06.395771 (+ 214us) server_negotiation.cc:407] Connection header received
0928 02:39:06.396771 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:06.396772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:06.396784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:06.396834 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:06.397474 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.397972 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.398954 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.399159 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.399239 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:06.401556 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:06.401557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:06.401558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:06.401565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:06.402919 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:06.403353 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:06.403354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:06.403354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:06.403413 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:06.403641 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:06.403642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:06.403642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:06.403741 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:06.403791 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:06.404103 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 02:39:06.404173 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:39:06.404342 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:06.395483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34521 (local address 127.9.130.3:49341)
0928 02:39:06.395643 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:06.395676 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:06.396651 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:06.396901 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:06.396905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:06.397151 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:06.397384 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:06.397389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.398058 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.398058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:06.398871 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:06.398873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.399247 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.399254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:06.399390 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:06.399942 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:06.399950 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:06.401455 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:06.403010 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:06.403016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:06.403018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:06.403272 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:06.403483 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:06.403483 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:06.403484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:06.403588 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:06.403792 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:06.403795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:06.403954 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 02:39:06.404035 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:39:06.404144 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260928 02:39:06.404788 13758 heartbeater.cc:347] Connected to a master server at 127.9.130.62:34521
I20260928 02:39:06.404987 13758 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:06.405190 13758 heartbeater.cc:511] Master 127.9.130.62:34521 requested a full tablet report, sending...
I20260928 02:39:06.405531 13299 ts_manager.cc:194] Registered new tserver with Master: a3301a26651a4a738c62cae9d01df586 (127.9.130.3:38813)
I20260928 02:39:06.405849 13299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:49341
I20260928 02:39:06.411602 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:39:06 dist-test-slave-kkrh krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:39:06 dist-test-slave-kkrh krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.1@KRBTEST.COM
I20260928 02:39:06.434271 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:06.426903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.9.130.1:45141)
0928 02:39:06.427063 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:06.427068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:06.427078 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:39:06.427103 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:06.427104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:06.427118 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:06.427184 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:06.427717 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.428491 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.429202 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.429522 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.429579 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:06.431681 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:06.431688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:06.431688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:06.431699 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:06.433098 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:06.433420 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:06.433423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:06.433424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:06.433492 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:06.433678 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:06.433679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:06.433679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:06.433817 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:06.433962 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:06.434009 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 02:39:06.434089 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
Sep 28 02:39:06 dist-test-slave-kkrh krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:06.442118 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:06.435495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.9.130.62:34521)
0928 02:39:06.435538 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:06.435539 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:06.435564 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:39:06.435615 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:06.435615 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:06.435624 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:06.435668 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:06.436149 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.436632 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.437321 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:06.437513 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:06.437566 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:06.439684 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:06.439686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:06.439686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:06.439692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:06.440937 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:06.441275 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:06.441276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:06.441276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:06.441341 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:06.441534 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:06.441534 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:06.441535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:06.441683 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:06.441773 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:06.441866 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 02:39:06.441932 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 02:39:06.442348 13299 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42830
I20260928 02:39:06.442662 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 13361
I20260928 02:39:06.446966 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 13495
I20260928 02:39:06.451117 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 13629
I20260928 02:39:06.455473 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 13267
2026-09-28T02:39:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1853 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:39:06 dist-test-slave-kkrh krb5kdc[13770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:39:06 dist-test-slave-kkrh krb5kdc[13770](info): set up 2 sockets
Sep 28 02:39:06 dist-test-slave-kkrh krb5kdc[13770](info): commencing operation
krb5kdc: starting...
W20260928 02:39:07.571923 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:39:07 dist-test-slave-kkrh krb5kdc[13770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:39:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:39:07.688560 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:42897
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:40531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:42897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:07.772657 13783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:07.772852 13783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:07.772876 13783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:07.774016 13783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:07.774060 13783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:07.774075 13783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:07.774111 13783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:07.774142 13783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:07.775655 13783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:40531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:42897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:42897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:07.775938 13783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:07.776172 13783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:07.780877 13789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:07.780997 13788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:07.780893 13791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:07.781123 13783 server_base.cc:1034] running on GCE node
I20260928 02:39:07.781363 13783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:07.781641 13783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:07.782799 13783 hybrid_clock.cc:648] HybridClock initialized: now 1790563147782777 us; error 55 us; skew 500 ppm
Sep 28 02:39:07 dist-test-slave-kkrh krb5kdc[13770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563147, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:07.785223 13783 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:39:07.785508 13783 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:07.785552 13783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:07.785640 13783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:07.786628 13783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "1b8f5916e25b43d08e829f2cb5bf065f"
format_stamp: "Formatted at 2026-09-28 02:39:07 on dist-test-slave-kkrh"
server_key: "33cc58b4c8ef3199f95c7568577becec"
server_key_iv: "22fe0965ac4811b4c49fe929549a2c2f"
server_key_version: "encryptionkey@0"
I20260928 02:39:07.787037 13783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "1b8f5916e25b43d08e829f2cb5bf065f"
format_stamp: "Formatted at 2026-09-28 02:39:07 on dist-test-slave-kkrh"
server_key: "33cc58b4c8ef3199f95c7568577becec"
server_key_iv: "22fe0965ac4811b4c49fe929549a2c2f"
server_key_version: "encryptionkey@0"
I20260928 02:39:07.788339 13783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:39:07.789122 13797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:07.789278 13783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:07.789372 13783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "1b8f5916e25b43d08e829f2cb5bf065f"
format_stamp: "Formatted at 2026-09-28 02:39:07 on dist-test-slave-kkrh"
server_key: "33cc58b4c8ef3199f95c7568577becec"
server_key_iv: "22fe0965ac4811b4c49fe929549a2c2f"
server_key_version: "encryptionkey@0"
I20260928 02:39:07.789448 13783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:07.803370 13783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:07.805245 13783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:07.805406 13783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:07.809290 13783 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:42897
I20260928 02:39:07.809331 13849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:42897 every 8 connection(s)
I20260928 02:39:07.809630 13783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:07.810516 13850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:07.812841 13850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f: Bootstrap starting.
I20260928 02:39:07.812831 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 13783
I20260928 02:39:07.812924 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:07.813105 9736 external_mini_cluster.cc:1428] Setting key 19e6729ee2c51bb3d3765f427d51c6c6
I20260928 02:39:07.813501 13850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:07.813876 13850 log.cc:824] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:07.814647 13850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f: No bootstrap required, opened a new log
I20260928 02:39:07.816994 13850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8f5916e25b43d08e829f2cb5bf065f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42897 } }
I20260928 02:39:07.817165 13850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:07.817198 13850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b8f5916e25b43d08e829f2cb5bf065f, State: Initialized, Role: FOLLOWER
I20260928 02:39:07.817329 13850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8f5916e25b43d08e829f2cb5bf065f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42897 } }
I20260928 02:39:07.817420 13850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:07.817453 13850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:07.817492 13850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:07.818221 13850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8f5916e25b43d08e829f2cb5bf065f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42897 } }
I20260928 02:39:07.818322 13850 leader_election.cc:302] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b8f5916e25b43d08e829f2cb5bf065f; no voters:
I20260928 02:39:07.818514 13850 leader_election.cc:288] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:07.818567 13855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:07.818732 13855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [term 1 LEADER]: Becoming Leader. State: Replica: 1b8f5916e25b43d08e829f2cb5bf065f, State: Running, Role: LEADER
I20260928 02:39:07.818774 13850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:07.818823 13855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8f5916e25b43d08e829f2cb5bf065f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42897 } }
I20260928 02:39:07.819160 13857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b8f5916e25b43d08e829f2cb5bf065f. Latest consensus state: current_term: 1 leader_uuid: "1b8f5916e25b43d08e829f2cb5bf065f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8f5916e25b43d08e829f2cb5bf065f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42897 } } }
I20260928 02:39:07.819247 13857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:07.819410 13856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b8f5916e25b43d08e829f2cb5bf065f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8f5916e25b43d08e829f2cb5bf065f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42897 } } }
I20260928 02:39:07.819474 13856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:07.819598 13862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:07.820086 13862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:39:07 dist-test-slave-kkrh krb5kdc[13770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:07.821635 13872 mvcc.cc:205] Tried to move back new op lower bound from 7334146653474742272 to 7334146653466198016. Current Snapshot: MvccSnapshot[applied={T|T < 7334146653474742272}]
I20260928 02:39:07.822129 13862 catalog_manager.cc:1347] Generated new cluster ID: 1eb923106833442bb9d62e866a08fe6b
I20260928 02:39:07.822175 13862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:39:07.826356 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:07.813599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37672 (local address 127.9.130.62:42897)
0928 02:39:07.813764 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:07.813772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:07.813799 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:39:07.814255 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:07.814262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:07.814310 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:07.814468 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:07.815258 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:07.816204 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:07.817400 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:07.818019 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:07.818140 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:07.821010 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:07.821014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:07.821014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:07.821026 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:07.824735 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:07.825181 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:07.825184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:07.825184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:07.825284 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:07.825491 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:07.825491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:07.825492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:07.825766 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:07.825902 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:07.826049 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:39:07.826174 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260928 02:39:07.830541 13862 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:07.831015 13862 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:07.840256 13862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f: Generated new TSK 0
I20260928 02:39:07.840435 13862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:39:07.868726 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42897
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:40531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:07.952173 13876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:07.952363 13876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:07.952384 13876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:07.953521 13876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:07.953559 13876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:07.953586 13876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:07.955067 13876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:40531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:07.955295 13876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:07.955515 13876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:07.960371 13882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:07.960448 13881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:07.960362 13884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:07.960762 13876 server_base.cc:1034] running on GCE node
I20260928 02:39:07.961007 13876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:07.961242 13876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:07.962421 13876 hybrid_clock.cc:648] HybridClock initialized: now 1790563147962410 us; error 31 us; skew 500 ppm
Sep 28 02:39:07 dist-test-slave-kkrh krb5kdc[13770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563147, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:07.965015 13876 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:39:07.965397 13876 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:07.965445 13876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:07.965581 13876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:07.966573 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "02f1a1bf732342f7bc92221b4aea139b"
format_stamp: "Formatted at 2026-09-28 02:39:07 on dist-test-slave-kkrh"
server_key: "a44f41e1e65f82adaa130b108a3eceeb"
server_key_iv: "3c8f89383adfdf5b3fac879e71d6e8ad"
server_key_version: "encryptionkey@0"
I20260928 02:39:07.966921 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "02f1a1bf732342f7bc92221b4aea139b"
format_stamp: "Formatted at 2026-09-28 02:39:07 on dist-test-slave-kkrh"
server_key: "a44f41e1e65f82adaa130b108a3eceeb"
server_key_iv: "3c8f89383adfdf5b3fac879e71d6e8ad"
server_key_version: "encryptionkey@0"
I20260928 02:39:07.968230 13876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:39:07.969074 13890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:07.969259 13876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:07.969405 13876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "02f1a1bf732342f7bc92221b4aea139b"
format_stamp: "Formatted at 2026-09-28 02:39:07 on dist-test-slave-kkrh"
server_key: "a44f41e1e65f82adaa130b108a3eceeb"
server_key_iv: "3c8f89383adfdf5b3fac879e71d6e8ad"
server_key_version: "encryptionkey@0"
I20260928 02:39:07.969482 13876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:07.992419 13876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:07.994700 13876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:07.994877 13876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:07.995131 13876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:07.995445 13876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:07.995496 13876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:07.995529 13876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:07.995558 13876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.001204 13876 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:42987
I20260928 02:39:08.001300 14003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:42987 every 8 connection(s)
I20260928 02:39:08.001550 13876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:39:08.003944 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 13876
I20260928 02:39:08.004055 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:39:08.004326 9736 external_mini_cluster.cc:1428] Setting key 8e656bcbcc75a8878039213aa014e4c1
Sep 28 02:39:08 dist-test-slave-kkrh krb5kdc[13770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563147, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:08.011349 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.002401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:35501 (local address 127.9.130.62:42897)
0928 02:39:08.002452 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.002454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.002555 (+ 101us) server_negotiation.cc:407] Connection header received
0928 02:39:08.003033 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.003034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.003043 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.003106 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:08.004242 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.004972 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.005995 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.006276 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.006333 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.008520 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:08.008522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:08.008522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:08.008529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:08.009986 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:08.010395 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:08.010396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:08.010397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:08.010457 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:08.010697 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:08.010698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:08.010698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:08.010814 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:08.010887 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:08.011156 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 02:39:08.011257 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 02:39:08.011543 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.002227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:42897 (local address 127.9.130.1:35501)
0928 02:39:08.002427 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:08.002470 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:08.002944 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:08.003185 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:08.003188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:08.003491 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:08.004118 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.004125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.005063 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.005064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.005899 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.005902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.006356 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.006357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.006522 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.007020 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:08.007030 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:08.008427 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:08.010083 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:08.010089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:08.010091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:08.010322 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:08.010519 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:08.010520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:08.010520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:08.010633 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:08.010877 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:08.010883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:08.011030 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 02:39:08.011137 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:39:08.011300 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260928 02:39:08.012184 14004 heartbeater.cc:347] Connected to a master server at 127.9.130.62:42897
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
I20260928 02:39:08.012286 14004 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:08.012452 14004 heartbeater.cc:511] Master 127.9.130.62:42897 requested a full tablet report, sending...
I20260928 02:39:08.013015 13814 ts_manager.cc:194] Registered new tserver with Master: 02f1a1bf732342f7bc92221b4aea139b (127.9.130.1:42987)
I20260928 02:39:08.013746 13814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:35501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:39:08.030177 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42897
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:40531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:08.113896 14009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:08.114125 14009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:08.114162 14009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:08.115336 14009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:08.115404 14009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:08.115450 14009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:08.117030 14009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:40531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:08.117275 14009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:08.117489 14009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:08.122380 14014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:08.122434 14017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:08.122434 14015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:08.122713 14009 server_base.cc:1034] running on GCE node
I20260928 02:39:08.122884 14009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:08.123062 14009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:08.124233 14009 hybrid_clock.cc:648] HybridClock initialized: now 1790563148124220 us; error 26 us; skew 500 ppm
Sep 28 02:39:08 dist-test-slave-kkrh krb5kdc[13770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563148, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:08.126749 14009 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:39:08.127074 14009 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:08.127120 14009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:08.127271 14009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:08.128402 14009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "d406d01b3ae743c88761861588cfbbe0"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "99a0a610411756f31556eb8f10e176e3"
server_key_iv: "92af51dc897cde089437b76e2905607f"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.128752 14009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "d406d01b3ae743c88761861588cfbbe0"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "99a0a610411756f31556eb8f10e176e3"
server_key_iv: "92af51dc897cde089437b76e2905607f"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.130012 14009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:08.130718 14023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.130915 14009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:08.130998 14009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "d406d01b3ae743c88761861588cfbbe0"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "99a0a610411756f31556eb8f10e176e3"
server_key_iv: "92af51dc897cde089437b76e2905607f"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.131048 14009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:08.143414 14009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:08.145471 14009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:08.145582 14009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:08.145754 14009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:08.146078 14009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:08.146122 14009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.146145 14009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:08.146190 14009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.151609 14009 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:37669
I20260928 02:39:08.151695 14136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:37669 every 8 connection(s)
I20260928 02:39:08.151958 14009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:39:08.154953 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 14009
I20260928 02:39:08.155076 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:39:08.155274 9736 external_mini_cluster.cc:1428] Setting key b38a8c3a6b3d7cd93f7cc1a53acb5cc9
Sep 28 02:39:08 dist-test-slave-kkrh krb5kdc[13770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563148, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:08.161756 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.152708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:52379 (local address 127.9.130.62:42897)
0928 02:39:08.152768 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.152770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.152985 (+ 215us) server_negotiation.cc:407] Connection header received
0928 02:39:08.153541 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.153542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.153554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.153605 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:08.154519 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.155024 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.156259 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.156467 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.156545 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.158824 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:08.158825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:08.158825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:08.158831 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:08.160275 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:08.160793 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:08.160794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:08.160794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:08.160842 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:08.161130 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:08.161130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:08.161131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:08.161224 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:08.161273 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:08.161631 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 02:39:08.161695 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:39:08.162132 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.152702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:42897 (local address 127.9.130.2:52379)
0928 02:39:08.152871 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:08.152908 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:08.153440 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:08.153683 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:08.153687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:08.153944 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:08.154418 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.154423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.155138 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.155140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.156168 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.156170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.156553 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.156554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.156679 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.157166 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:08.157174 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:08.158710 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:08.160351 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:08.160358 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:08.160361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:08.160715 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:08.160914 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:08.160915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:08.160916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:08.161063 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:08.161289 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:08.161295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:08.161483 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 02:39:08.161622 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:39:08.161816 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260928 02:39:08.162648 14137 heartbeater.cc:347] Connected to a master server at 127.9.130.62:42897
I20260928 02:39:08.162748 14137 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:08.162890 14137 heartbeater.cc:511] Master 127.9.130.62:42897 requested a full tablet report, sending...
I20260928 02:39:08.163260 13814 ts_manager.cc:194] Registered new tserver with Master: d406d01b3ae743c88761861588cfbbe0 (127.9.130.2:37669)
I20260928 02:39:08.163674 13814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:52379
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:39:08.183580 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42897
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:40531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:08.267338 14142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:08.267546 14142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:08.267570 14142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:08.268676 14142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:08.268709 14142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:08.268735 14142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:08.270125 14142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:40531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:08.270298 14142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:08.270473 14142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:39:08.275548 14142 server_base.cc:1034] running on GCE node
W20260928 02:39:08.275520 14147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:08.275557 14150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:08.275521 14148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:08.275935 14142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:08.276126 14142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:08.277282 14142 hybrid_clock.cc:648] HybridClock initialized: now 1790563148277269 us; error 30 us; skew 500 ppm
Sep 28 02:39:08 dist-test-slave-kkrh krb5kdc[13770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563148, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:08.279790 14142 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:39:08.280109 14142 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:08.280153 14142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:08.280239 14142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:08.281193 14142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "85485605d4b345c4886e9eb57d5697af"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "6998fbe2ccab1dbbbe203bf41a21f73a"
server_key_iv: "a62f85dcbe759eeadb90d21ddf425373"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.281500 14142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "85485605d4b345c4886e9eb57d5697af"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "6998fbe2ccab1dbbbe203bf41a21f73a"
server_key_iv: "a62f85dcbe759eeadb90d21ddf425373"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.282662 14142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:08.283267 14156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.283417 14142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:08.283514 14142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "85485605d4b345c4886e9eb57d5697af"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "6998fbe2ccab1dbbbe203bf41a21f73a"
server_key_iv: "a62f85dcbe759eeadb90d21ddf425373"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.283583 14142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:08.295317 14142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:08.297634 14142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:08.297792 14142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:08.298022 14142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:08.298334 14142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:08.298384 14142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.298424 14142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:08.298452 14142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.304214 14142 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:37783
I20260928 02:39:08.304260 14269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:37783 every 8 connection(s)
I20260928 02:39:08.304538 14142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:39:08.308516 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 14142
I20260928 02:39:08.308636 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:39:08.308862 9736 external_mini_cluster.cc:1428] Setting key 43b2d1c8e6813791940a11de300bdd10
Sep 28 02:39:08 dist-test-slave-kkrh krb5kdc[13770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563148, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:08.313683 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.305231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:34339 (local address 127.9.130.62:42897)
0928 02:39:08.305292 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.305294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.305506 (+ 212us) server_negotiation.cc:407] Connection header received
0928 02:39:08.306020 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.306021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.306031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.306084 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:08.307010 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.307488 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.308787 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.309024 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.309110 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.311147 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:08.311149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:08.311149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:08.311156 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:08.312369 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:08.312762 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:08.312763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:08.312763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:08.312820 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:08.313059 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:08.313060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:08.313060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:08.313168 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:08.313242 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:08.313515 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 02:39:08.313592 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:39:08.313859 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.305237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:42897 (local address 127.9.130.3:34339)
0928 02:39:08.305393 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:08.305433 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:08.305933 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:08.306140 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:08.306143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:08.306406 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:08.306933 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.306939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.307569 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.307570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.308690 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.308693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.309118 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.309119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.309268 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.309774 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:08.309784 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:08.311059 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:08.312467 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:08.312472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:08.312474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:08.312678 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:08.312882 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:08.312883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:08.312883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:08.312995 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:08.313232 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:08.313237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:08.313375 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 02:39:08.313494 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 02:39:08.313642 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260928 02:39:08.314311 14270 heartbeater.cc:347] Connected to a master server at 127.9.130.62:42897
I20260928 02:39:08.314411 14270 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:08.314579 14270 heartbeater.cc:511] Master 127.9.130.62:42897 requested a full tablet report, sending...
I20260928 02:39:08.314980 13814 ts_manager.cc:194] Registered new tserver with Master: 85485605d4b345c4886e9eb57d5697af (127.9.130.3:37783)
I20260928 02:39:08.315382 13814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:34339
I20260928 02:39:08.320374 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:39:08.326660 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.321634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.9.130.62:42897)
0928 02:39:08.321686 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.321688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.321780 (+ 92us) server_negotiation.cc:407] Connection header received
0928 02:39:08.321842 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.321843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.321851 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.321890 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:08.322405 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.322843 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.323487 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.323650 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.323696 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.324740 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:08.324741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:08.324741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:08.324747 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:08.325806 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:08.326089 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:08.326090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:08.326090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:08.326136 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:08.326283 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:08.326283 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:08.326284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:08.326386 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:08.326455 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:08.326555 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 02:39:08.326605 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:39:08.328074 13814 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:39:08.329574 13814 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:39:08.335649 14282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.331550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:42987 (local address 127.0.0.1:38198)
0928 02:39:08.331883 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:08.331892 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:08.331940 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:08.332194 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:08.332197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:08.332200 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:08.332269 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.332271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.333329 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.333330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.334447 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.334449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.335345 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.335346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.335429 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.335443 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 02:39:08.335524 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:39:08.335571 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":46,"threads_started":1}
I20260928 02:39:08.335753 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.331634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.9.130.2:37669)
0928 02:39:08.331785 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.331790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.331924 (+ 134us) server_negotiation.cc:407] Connection header received
0928 02:39:08.332084 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.332085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.332098 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.332193 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:08.332320 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.333057 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.334584 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.335261 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.335334 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.335392 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:08.335623 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 02:39:08.335665 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260928 02:39:08.335990 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.331533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:37669 (local address 127.0.0.1:40214)
0928 02:39:08.331805 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:08.331863 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:08.331940 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:08.332193 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:08.332196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:08.332198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:08.332258 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.332260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.333140 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.333141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.334452 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.334455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.335435 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.335436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.335526 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.335531 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:39:08.335882 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 02:39:08.335923 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":61,"threads_started":1}
I20260928 02:39:08.335683 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.331656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.9.130.1:42987)
0928 02:39:08.331859 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.331862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.332029 (+ 167us) server_negotiation.cc:407] Connection header received
0928 02:39:08.332078 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.332081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.332101 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.332190 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:08.332446 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.333231 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.334518 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.335259 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.335335 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.335392 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:08.335549 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:39:08.335601 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
I20260928 02:39:08.336714 13938 tablet_service.cc:1467] Processing CreateTablet for tablet a0fa0e76dcda417c810eafa8ef70e290 (DEFAULT_TABLE table=test-table [id=1dbbfe60cbf84c849556bc78fc3f4383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:08.336716 14071 tablet_service.cc:1467] Processing CreateTablet for tablet a0fa0e76dcda417c810eafa8ef70e290 (DEFAULT_TABLE table=test-table [id=1dbbfe60cbf84c849556bc78fc3f4383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:08.337011 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fa0e76dcda417c810eafa8ef70e290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:08.337074 14071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fa0e76dcda417c810eafa8ef70e290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:08.337915 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.331733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:37783 (local address 127.0.0.1:39866)
0928 02:39:08.333358 (+ 1625us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:08.333368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:08.333419 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:08.333660 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:08.333661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:08.333663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:08.333695 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.333697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.334856 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.334857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.335691 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.335694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.337700 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.337701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.337762 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.337766 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:39:08.337814 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:39:08.337851 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1612,"thread_start_us":31,"threads_started":1}
I20260928 02:39:08.338205 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.331874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.9.130.3:37783)
0928 02:39:08.332004 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.332008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.333416 (+ 1408us) server_negotiation.cc:407] Connection header received
0928 02:39:08.333489 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.333492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.333509 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.333584 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:08.333774 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.334773 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.335782 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.337545 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.337610 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.337669 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:08.338086 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 02:39:08.338128 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":84,"threads_started":1}
I20260928 02:39:08.338865 14204 tablet_service.cc:1467] Processing CreateTablet for tablet a0fa0e76dcda417c810eafa8ef70e290 (DEFAULT_TABLE table=test-table [id=1dbbfe60cbf84c849556bc78fc3f4383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:08.339180 14204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fa0e76dcda417c810eafa8ef70e290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:08.339969 14284 tablet_bootstrap.cc:492] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b: Bootstrap starting.
I20260928 02:39:08.340328 14285 tablet_bootstrap.cc:492] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0: Bootstrap starting.
I20260928 02:39:08.340560 14284 tablet_bootstrap.cc:654] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:08.340965 14285 tablet_bootstrap.cc:654] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:08.341321 14284 log.cc:824] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:08.341578 14285 log.cc:824] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:08.342099 14284 tablet_bootstrap.cc:492] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b: No bootstrap required, opened a new log
I20260928 02:39:08.342171 14284 ts_tablet_manager.cc:1397] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:39:08.342250 14285 tablet_bootstrap.cc:492] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0: No bootstrap required, opened a new log
I20260928 02:39:08.342312 14285 ts_tablet_manager.cc:1397] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:39:08.342502 14288 tablet_bootstrap.cc:492] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af: Bootstrap starting.
I20260928 02:39:08.343122 14288 tablet_bootstrap.cc:654] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:08.343739 14288 log.cc:824] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:08.344106 14285 raft_consensus.cc:348] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406d01b3ae743c88761861588cfbbe0" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37669 } } peers { permanent_uuid: "85485605d4b345c4886e9eb57d5697af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 37783 } } peers { permanent_uuid: "02f1a1bf732342f7bc92221b4aea139b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42987 } }
I20260928 02:39:08.344266 14285 raft_consensus.cc:374] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:08.344309 14285 raft_consensus.cc:729] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d406d01b3ae743c88761861588cfbbe0, State: Initialized, Role: FOLLOWER
I20260928 02:39:08.344441 14285 consensus_queue.cc:260] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406d01b3ae743c88761861588cfbbe0" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37669 } } peers { permanent_uuid: "85485605d4b345c4886e9eb57d5697af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 37783 } } peers { permanent_uuid: "02f1a1bf732342f7bc92221b4aea139b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42987 } }
I20260928 02:39:08.344364 14284 raft_consensus.cc:348] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406d01b3ae743c88761861588cfbbe0" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37669 } } peers { permanent_uuid: "85485605d4b345c4886e9eb57d5697af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 37783 } } peers { permanent_uuid: "02f1a1bf732342f7bc92221b4aea139b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42987 } }
I20260928 02:39:08.344511 14288 tablet_bootstrap.cc:492] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af: No bootstrap required, opened a new log
I20260928 02:39:08.344519 14284 raft_consensus.cc:374] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:08.344566 14288 ts_tablet_manager.cc:1397] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:39:08.344566 14284 raft_consensus.cc:729] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f1a1bf732342f7bc92221b4aea139b, State: Initialized, Role: FOLLOWER
I20260928 02:39:08.344674 14284 consensus_queue.cc:260] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406d01b3ae743c88761861588cfbbe0" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37669 } } peers { permanent_uuid: "85485605d4b345c4886e9eb57d5697af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 37783 } } peers { permanent_uuid: "02f1a1bf732342f7bc92221b4aea139b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42987 } }
I20260928 02:39:08.344722 14285 ts_tablet_manager.cc:1428] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:39:08.344830 14137 heartbeater.cc:503] Master 127.9.130.62:42897 was elected leader, sending a full tablet report...
I20260928 02:39:08.344929 14284 ts_tablet_manager.cc:1428] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:39:08.344947 14004 heartbeater.cc:503] Master 127.9.130.62:42897 was elected leader, sending a full tablet report...
I20260928 02:39:08.346370 14288 raft_consensus.cc:348] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406d01b3ae743c88761861588cfbbe0" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37669 } } peers { permanent_uuid: "85485605d4b345c4886e9eb57d5697af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 37783 } } peers { permanent_uuid: "02f1a1bf732342f7bc92221b4aea139b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42987 } }
I20260928 02:39:08.346484 14288 raft_consensus.cc:374] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:08.346505 14288 raft_consensus.cc:729] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85485605d4b345c4886e9eb57d5697af, State: Initialized, Role: FOLLOWER
I20260928 02:39:08.346575 14288 consensus_queue.cc:260] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406d01b3ae743c88761861588cfbbe0" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37669 } } peers { permanent_uuid: "85485605d4b345c4886e9eb57d5697af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 37783 } } peers { permanent_uuid: "02f1a1bf732342f7bc92221b4aea139b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42987 } }
I20260928 02:39:08.346750 14288 ts_tablet_manager.cc:1428] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:39:08.346925 14270 heartbeater.cc:503] Master 127.9.130.62:42897 was elected leader, sending a full tablet report...
W20260928 02:39:08.402483 14138 tablet.cc:2369] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:39:08.439249 14291 raft_consensus.cc:482] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:39:08.439392 14291 raft_consensus.cc:504] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406d01b3ae743c88761861588cfbbe0" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37669 } } peers { permanent_uuid: "85485605d4b345c4886e9eb57d5697af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 37783 } } peers { permanent_uuid: "02f1a1bf732342f7bc92221b4aea139b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42987 } }
I20260928 02:39:08.439791 14291 leader_election.cc:288] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d406d01b3ae743c88761861588cfbbe0 (127.9.130.2:37669), 85485605d4b345c4886e9eb57d5697af (127.9.130.3:37783)
I20260928 02:39:08.442750 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.440076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:57261 (local address 127.9.130.2:37669)
0928 02:39:08.440139 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.440141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.440180 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:39:08.440214 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.440215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.440239 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.440300 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:08.440497 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.441031 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.441990 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.442388 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.442442 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.442449 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:08.442631 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 02:39:08.442678 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:39:08.442756 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.439996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:37669 (local address 127.9.130.1:57261)
0928 02:39:08.440053 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:08.440066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:08.440114 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:08.440306 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:08.440307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:08.440309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:08.440404 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.440406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.441106 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.441107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.441886 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.441888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.442477 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.442478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.442558 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.442562 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:39:08.442616 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:39:08.442643 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 02:39:08.442802 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.440214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:36417 (local address 127.9.130.3:37783)
0928 02:39:08.440260 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.440262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.440385 (+ 123us) server_negotiation.cc:407] Connection header received
0928 02:39:08.440418 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.440420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.440432 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.440488 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:08.440667 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.441140 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.442071 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.442453 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.442525 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.442535 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:08.442690 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 02:39:08.442732 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:39:08.442858 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.440113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:37783 (local address 127.9.130.1:36417)
0928 02:39:08.440306 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:08.440317 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:08.440374 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:08.440531 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:08.440531 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:08.440533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:08.440599 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.440601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.441216 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.441216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.441987 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:08.441989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.442537 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.442537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:08.442610 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.442614 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:39:08.442670 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:39:08.442702 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":116,"threads_started":1}
I20260928 02:39:08.443117 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa0e76dcda417c810eafa8ef70e290" candidate_uuid: "02f1a1bf732342f7bc92221b4aea139b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85485605d4b345c4886e9eb57d5697af" is_pre_election: true
I20260928 02:39:08.443118 14091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa0e76dcda417c810eafa8ef70e290" candidate_uuid: "02f1a1bf732342f7bc92221b4aea139b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d406d01b3ae743c88761861588cfbbe0" is_pre_election: true
I20260928 02:39:08.443262 14091 raft_consensus.cc:2454] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f1a1bf732342f7bc92221b4aea139b in term 0.
I20260928 02:39:08.443262 14224 raft_consensus.cc:2454] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f1a1bf732342f7bc92221b4aea139b in term 0.
I20260928 02:39:08.443495 13893 leader_election.cc:302] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f1a1bf732342f7bc92221b4aea139b, 85485605d4b345c4886e9eb57d5697af; no voters:
I20260928 02:39:08.443615 14291 raft_consensus.cc:2781] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:39:08.443698 14291 raft_consensus.cc:482] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:39:08.443732 14291 raft_consensus.cc:3037] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:08.444379 14291 raft_consensus.cc:504] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406d01b3ae743c88761861588cfbbe0" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37669 } } peers { permanent_uuid: "85485605d4b345c4886e9eb57d5697af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 37783 } } peers { permanent_uuid: "02f1a1bf732342f7bc92221b4aea139b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42987 } }
I20260928 02:39:08.444514 14291 leader_election.cc:288] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [CANDIDATE]: Term 1 election: Requested vote from peers d406d01b3ae743c88761861588cfbbe0 (127.9.130.2:37669), 85485605d4b345c4886e9eb57d5697af (127.9.130.3:37783)
I20260928 02:39:08.444679 14091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa0e76dcda417c810eafa8ef70e290" candidate_uuid: "02f1a1bf732342f7bc92221b4aea139b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d406d01b3ae743c88761861588cfbbe0"
I20260928 02:39:08.444720 14224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa0e76dcda417c810eafa8ef70e290" candidate_uuid: "02f1a1bf732342f7bc92221b4aea139b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85485605d4b345c4886e9eb57d5697af"
I20260928 02:39:08.444752 14091 raft_consensus.cc:3037] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:08.444785 14224 raft_consensus.cc:3037] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:08.445613 14224 raft_consensus.cc:2454] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f1a1bf732342f7bc92221b4aea139b in term 1.
I20260928 02:39:08.445631 14091 raft_consensus.cc:2454] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f1a1bf732342f7bc92221b4aea139b in term 1.
I20260928 02:39:08.445770 13893 leader_election.cc:302] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f1a1bf732342f7bc92221b4aea139b, d406d01b3ae743c88761861588cfbbe0; no voters:
I20260928 02:39:08.445847 14291 raft_consensus.cc:2781] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:08.445993 14291 raft_consensus.cc:686] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [term 1 LEADER]: Becoming Leader. State: Replica: 02f1a1bf732342f7bc92221b4aea139b, State: Running, Role: LEADER
I20260928 02:39:08.446095 14291 consensus_queue.cc:237] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406d01b3ae743c88761861588cfbbe0" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37669 } } peers { permanent_uuid: "85485605d4b345c4886e9eb57d5697af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 37783 } } peers { permanent_uuid: "02f1a1bf732342f7bc92221b4aea139b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42987 } }
I20260928 02:39:08.446821 13814 catalog_manager.cc:5495] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b reported cstate change: term changed from 0 to 1, leader changed from <none> to 02f1a1bf732342f7bc92221b4aea139b (127.9.130.1). New cstate: current_term: 1 leader_uuid: "02f1a1bf732342f7bc92221b4aea139b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406d01b3ae743c88761861588cfbbe0" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85485605d4b345c4886e9eb57d5697af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 37783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02f1a1bf732342f7bc92221b4aea139b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42987 } health_report { overall_health: HEALTHY } } }
I20260928 02:39:08.478786 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.476666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.9.130.1:42987)
0928 02:39:08.476717 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.476719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.476727 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:08.476760 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.476761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.476778 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.476839 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:39:08.476938 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.477390 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.478099 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:08.478271 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:08.478319 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:08.478442 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:39:08.478556 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:39:08.478615 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:08.478699 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:39:08.478715 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:39:08.480490 14224 raft_consensus.cc:1261] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af [term 1 FOLLOWER]: Refusing update from remote peer 02f1a1bf732342f7bc92221b4aea139b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:08.480523 14091 raft_consensus.cc:1261] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0 [term 1 FOLLOWER]: Refusing update from remote peer 02f1a1bf732342f7bc92221b4aea139b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:08.480836 14291 consensus_queue.cc:1035] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d406d01b3ae743c88761861588cfbbe0" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:08.480970 14294 consensus_queue.cc:1035] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85485605d4b345c4886e9eb57d5697af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 37783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:08.483768 14298 mvcc.cc:205] Tried to move back new op lower bound from 7334146656173473792 to 7334146656035577856. Current Snapshot: MvccSnapshot[applied={T|T < 7334146656173473792}]
I20260928 02:39:08.485391 13813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:08.485478 13813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:08.488507 13813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b8f5916e25b43d08e829f2cb5bf065f: Sending DeleteTablet for 3 replicas of tablet a0fa0e76dcda417c810eafa8ef70e290
I20260928 02:39:08.488906 14204 tablet_service.cc:1514] Processing DeleteTablet for tablet a0fa0e76dcda417c810eafa8ef70e290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:08 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:39866
I20260928 02:39:08.490114 14071 tablet_service.cc:1514] Processing DeleteTablet for tablet a0fa0e76dcda417c810eafa8ef70e290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:08 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:40214
I20260928 02:39:08.490520 13938 tablet_service.cc:1514] Processing DeleteTablet for tablet a0fa0e76dcda417c810eafa8ef70e290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:08 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:38198
I20260928 02:39:08.490715 14308 tablet_replica.cc:330] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b: stopping tablet replica
I20260928 02:39:08.490762 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 13876
I20260928 02:39:08.490803 14308 raft_consensus.cc:2229] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:39:08.490919 14308 raft_consensus.cc:2258] T a0fa0e76dcda417c810eafa8ef70e290 P 02f1a1bf732342f7bc92221b4aea139b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:08.491389 14306 tablet_replica.cc:330] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af: stopping tablet replica
I20260928 02:39:08.491556 14306 raft_consensus.cc:2229] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:08.491669 14306 raft_consensus.cc:2258] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:08.493516 14306 ts_tablet_manager.cc:1905] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:08.493093 14307 tablet_replica.cc:330] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0: stopping tablet replica
I20260928 02:39:08.494010 14307 raft_consensus.cc:2229] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:08.494140 14307 raft_consensus.cc:2258] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:08.494613 14307 ts_tablet_manager.cc:1905] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:08.495256 14306 ts_tablet_manager.cc:1918] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:08.495321 14306 log.cc:1196] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/a0fa0e76dcda417c810eafa8ef70e290
I20260928 02:39:08.495617 14306 ts_tablet_manager.cc:1939] T a0fa0e76dcda417c810eafa8ef70e290 P 85485605d4b345c4886e9eb57d5697af: Deleting consensus metadata
I20260928 02:39:08.496097 13800 catalog_manager.cc:4841] TS 85485605d4b345c4886e9eb57d5697af (127.9.130.3:37783): tablet a0fa0e76dcda417c810eafa8ef70e290 (table test-table [id=1dbbfe60cbf84c849556bc78fc3f4383]) successfully deleted
I20260928 02:39:08.496520 14307 ts_tablet_manager.cc:1918] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:08.496608 14307 log.cc:1196] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/a0fa0e76dcda417c810eafa8ef70e290
I20260928 02:39:08.496869 14307 ts_tablet_manager.cc:1939] T a0fa0e76dcda417c810eafa8ef70e290 P d406d01b3ae743c88761861588cfbbe0: Deleting consensus metadata
I20260928 02:39:08.497296 13800 catalog_manager.cc:4841] TS d406d01b3ae743c88761861588cfbbe0 (127.9.130.2:37669): tablet a0fa0e76dcda417c810eafa8ef70e290 (table test-table [id=1dbbfe60cbf84c849556bc78fc3f4383]) successfully deleted
W20260928 02:39:08.497620 13801 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:42987 (error 108)
I20260928 02:39:08.497871 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 14009
I20260928 02:39:08.498524 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.498051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:42987 (local address 127.0.0.1:38214)
0928 02:39:08.498433 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:08.498477 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358}
W20260928 02:39:08.498870 13801 catalog_manager.cc:4584] TS 02f1a1bf732342f7bc92221b4aea139b (127.9.130.1:42987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a0fa0e76dcda417c810eafa8ef70e290: Network error: Client connection negotiation failed: client connection to 127.9.130.1:42987: connect: Connection refused (error 111)
I20260928 02:39:08.502460 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 14142
I20260928 02:39:08.506525 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 13783
2026-09-28T02:39:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2050 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T02:39:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:08Z Disabled control of system clock
I20260928 02:39:08.533764 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:41463
--webserver_interface=127.9.130.62
--webserver_port=0
--builtin_ntp_servers=127.9.130.20:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:41463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:39:08.612696 14313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:08.612933 14313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:08.612972 14313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:08.614097 14313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:08.614161 14313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:08.614181 14313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:08.614203 14313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:08.614229 14313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:08.615562 14313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:41463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:41463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:08.615849 14313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:08.616145 14313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:08.620965 14318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:08.620981 14321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:08.621200 14313 server_base.cc:1034] running on GCE node
W20260928 02:39:08.620955 14319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:08.621477 14313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:08.621722 14313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:08.622869 14313 hybrid_clock.cc:648] HybridClock initialized: now 1790563148622855 us; error 30 us; skew 500 ppm
I20260928 02:39:08.624135 14313 webserver.cc:460] Webserver started at http://127.9.130.62:42039/ using document root <none> and password file <none>
I20260928 02:39:08.624356 14313 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:08.624419 14313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:08.624568 14313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:08.625635 14313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "49f245d3ddfb43b5a2531dc9394da43b"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "a2660d5f38b4173ad1dc3059edd0393a"
server_key_iv: "6cd6800883e2ab854193552dd2d8e21c"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.625981 14313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "49f245d3ddfb43b5a2531dc9394da43b"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "a2660d5f38b4173ad1dc3059edd0393a"
server_key_iv: "6cd6800883e2ab854193552dd2d8e21c"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.627223 14313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:39:08.627939 14327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.628111 14313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:08.628202 14313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "49f245d3ddfb43b5a2531dc9394da43b"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "a2660d5f38b4173ad1dc3059edd0393a"
server_key_iv: "6cd6800883e2ab854193552dd2d8e21c"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.628280 14313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:08.631495 14313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:08.631743 14313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:08.631899 14313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:08.635541 14313 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:41463
I20260928 02:39:08.635568 14379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:41463 every 8 connection(s)
I20260928 02:39:08.635933 14313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:08.636597 14380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:08.637766 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 14313
I20260928 02:39:08.637868 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:08.638087 9736 external_mini_cluster.cc:1428] Setting key 884c2775129e3d10fbf61a73c7fa1310
I20260928 02:39:08.640151 14380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b: Bootstrap starting.
I20260928 02:39:08.640524 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.638599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.9.130.62:41463)
0928 02:39:08.638767 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.638773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.638797 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:39:08.639217 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.639224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.639262 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.639371 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:08.640085 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:08.640088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:08.640089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:08.640093 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:08.640133 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:08.640142 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:08.640190 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:08.640275 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:39:08.640332 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260928 02:39:08.640841 14380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:08.641155 14380 log.cc:824] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:08.641793 14380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b: No bootstrap required, opened a new log
I20260928 02:39:08.643096 14380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f245d3ddfb43b5a2531dc9394da43b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41463 } }
I20260928 02:39:08.643220 14380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:08.643261 14380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f245d3ddfb43b5a2531dc9394da43b, State: Initialized, Role: FOLLOWER
I20260928 02:39:08.643374 14380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f245d3ddfb43b5a2531dc9394da43b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41463 } }
I20260928 02:39:08.643455 14380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:08.643514 14380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:08.643563 14380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:08.644106 14380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f245d3ddfb43b5a2531dc9394da43b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41463 } }
I20260928 02:39:08.644209 14380 leader_election.cc:302] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49f245d3ddfb43b5a2531dc9394da43b; no voters:
I20260928 02:39:08.644366 14380 leader_election.cc:288] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:08.644622 14380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:08.644647 14385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:08.644775 14385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [term 1 LEADER]: Becoming Leader. State: Replica: 49f245d3ddfb43b5a2531dc9394da43b, State: Running, Role: LEADER
I20260928 02:39:08.644884 14385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f245d3ddfb43b5a2531dc9394da43b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41463 } }
I20260928 02:39:08.645306 14387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49f245d3ddfb43b5a2531dc9394da43b. Latest consensus state: current_term: 1 leader_uuid: "49f245d3ddfb43b5a2531dc9394da43b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f245d3ddfb43b5a2531dc9394da43b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41463 } } }
I20260928 02:39:08.645427 14387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:08.645754 14386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49f245d3ddfb43b5a2531dc9394da43b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f245d3ddfb43b5a2531dc9394da43b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41463 } } }
I20260928 02:39:08.645841 14386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:08.645939 14394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:08.646456 14394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:08.647921 14394 catalog_manager.cc:1347] Generated new cluster ID: 1c23be494932493fa56dd5a9a95dd7c1
I20260928 02:39:08.647985 14394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:39:08.654614 14394 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:08.654772 14394 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:08.657236 14394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b: Generated new TSK 0
I20260928 02:39:08.657404 14394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:39:08.661187 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41463
--builtin_ntp_servers=127.9.130.20:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:39:08.740433 14404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:08.740657 14404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:08.740695 14404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:08.741865 14404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:08.741925 14404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:08.741971 14404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:08.743333 14404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:08.743584 14404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:08.743906 14404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:08.748764 14412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:08.748771 14409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:08.748986 14410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:08.748972 14404 server_base.cc:1034] running on GCE node
I20260928 02:39:08.749238 14404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:08.749490 14404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:08.750633 14404 hybrid_clock.cc:648] HybridClock initialized: now 1790563148750614 us; error 35 us; skew 500 ppm
I20260928 02:39:08.751899 14404 webserver.cc:460] Webserver started at http://127.9.130.1:34851/ using document root <none> and password file <none>
I20260928 02:39:08.752158 14404 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:08.752202 14404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:08.752339 14404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:08.753405 14404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "0fd40f5e7b77d0a34b1372e343957c11"
server_key_iv: "051f35e921b55808df9e6c05e635683f"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.753743 14404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "0fd40f5e7b77d0a34b1372e343957c11"
server_key_iv: "051f35e921b55808df9e6c05e635683f"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.754992 14404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:39:08.755718 14418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.755880 14404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:08.755975 14404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "0fd40f5e7b77d0a34b1372e343957c11"
server_key_iv: "051f35e921b55808df9e6c05e635683f"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.756049 14404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:08.759217 14404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:08.759466 14404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:08.759591 14404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:08.759802 14404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:08.760123 14404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:08.760154 14404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.760192 14404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:08.760217 14404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.767082 14404 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:33823
I20260928 02:39:08.767113 14531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:33823 every 8 connection(s)
I20260928 02:39:08.767421 14404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:39:08.770263 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.768110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:39091 (local address 127.9.130.62:41463)
0928 02:39:08.768174 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.768176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.768391 (+ 215us) server_negotiation.cc:407] Connection header received
0928 02:39:08.768847 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.768848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.768854 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.768933 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:08.770066 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:08.770068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:08.770068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:08.770072 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:08.770089 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:08.770093 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:08.770147 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:08.770200 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 02:39:08.770211 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:39:08.770466 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.768098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:41463 (local address 127.9.130.1:39091)
0928 02:39:08.768267 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:08.768307 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:08.768757 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:08.768984 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:08.768988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:08.768995 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:08.769920 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:08.769921 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:08.769949 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:08.769977 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:08.770152 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:08.770155 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:08.770157 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:39:08.770206 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:39:08.770250 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260928 02:39:08.770929 14532 heartbeater.cc:347] Connected to a master server at 127.9.130.62:41463
I20260928 02:39:08.771019 14532 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:08.771162 14532 heartbeater.cc:511] Master 127.9.130.62:41463 requested a full tablet report, sending...
I20260928 02:39:08.771605 14344 ts_manager.cc:194] Registered new tserver with Master: cb6fa0a0a44c40d68f140c3c3cdacdf1 (127.9.130.1:33823)
I20260928 02:39:08.775909 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 14404
I20260928 02:39:08.776036 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:39:08.776219 9736 external_mini_cluster.cc:1428] Setting key 25fe2574515dfa89613958c969bf563b
I20260928 02:39:08.777104 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41463
--builtin_ntp_servers=127.9.130.20:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:39:08.856678 14535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:08.856894 14535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:08.856916 14535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:08.858093 14535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:08.858132 14535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:08.858187 14535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:08.859696 14535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:08.859958 14535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:08.860260 14535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:08.865123 14541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:08.865127 14540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:08.865319 14543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:08.865221 14535 server_base.cc:1034] running on GCE node
I20260928 02:39:08.865614 14535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:08.865863 14535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:08.867057 14535 hybrid_clock.cc:648] HybridClock initialized: now 1790563148867042 us; error 42 us; skew 500 ppm
I20260928 02:39:08.868469 14535 webserver.cc:460] Webserver started at http://127.9.130.2:45479/ using document root <none> and password file <none>
I20260928 02:39:08.868707 14535 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:08.868773 14535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:08.868932 14535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:08.870005 14535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "9f331afb720345ceba101620c028ae16"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "ea8c6bba209abd18b6fb96d713170769"
server_key_iv: "c4d226c71e7e20817f34659ccdb670a8"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.870373 14535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "9f331afb720345ceba101620c028ae16"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "ea8c6bba209abd18b6fb96d713170769"
server_key_iv: "c4d226c71e7e20817f34659ccdb670a8"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.871673 14535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:39:08.872545 14549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.872754 14535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:08.872845 14535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "9f331afb720345ceba101620c028ae16"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "ea8c6bba209abd18b6fb96d713170769"
server_key_iv: "c4d226c71e7e20817f34659ccdb670a8"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.872925 14535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:08.876266 14535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:08.876514 14535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:08.876652 14535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:08.876847 14535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:08.877146 14535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:08.877194 14535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.877233 14535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:08.877259 14535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.883008 14535 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:39613
I20260928 02:39:08.883042 14662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:39613 every 8 connection(s)
I20260928 02:39:08.883378 14535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:39:08.886409 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.884151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:37699 (local address 127.9.130.62:41463)
0928 02:39:08.884204 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.884206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.884419 (+ 213us) server_negotiation.cc:407] Connection header received
0928 02:39:08.884964 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.884965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.884972 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.885032 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:08.886180 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:08.886181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:08.886181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:08.886186 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:08.886201 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:08.886205 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:08.886252 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:08.886325 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 02:39:08.886340 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:39:08.886602 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.884134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:41463 (local address 127.9.130.2:37699)
0928 02:39:08.884311 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:08.884341 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:08.884844 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:08.885110 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:08.885114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:08.885123 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:08.886033 (+ 910us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:08.886033 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:08.886060 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:08.886086 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:08.886250 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:08.886254 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:08.886257 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:08.886334 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:39:08.886391 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260928 02:39:08.887120 14663 heartbeater.cc:347] Connected to a master server at 127.9.130.62:41463
I20260928 02:39:08.887192 14663 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:08.887343 14663 heartbeater.cc:511] Master 127.9.130.62:41463 requested a full tablet report, sending...
I20260928 02:39:08.887723 14344 ts_manager.cc:194] Registered new tserver with Master: 9f331afb720345ceba101620c028ae16 (127.9.130.2:39613)
I20260928 02:39:08.891429 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 14535
I20260928 02:39:08.891510 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:39:08.891702 9736 external_mini_cluster.cc:1428] Setting key c0a641900ab097329cd1bcfd393d2d43
I20260928 02:39:08.892656 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41463
--builtin_ntp_servers=127.9.130.20:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:39:08.971309 14666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:08.971493 14666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:08.971513 14666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:08.972666 14666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:08.972704 14666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:08.972728 14666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:08.974045 14666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:08.974287 14666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:08.974570 14666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:08.979867 14674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:08.979897 14672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:08.979928 14666 server_base.cc:1034] running on GCE node
W20260928 02:39:08.979879 14671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:08.980350 14666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:08.980624 14666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:08.981763 14666 hybrid_clock.cc:648] HybridClock initialized: now 1790563148981755 us; error 36 us; skew 500 ppm
I20260928 02:39:08.982988 14666 webserver.cc:460] Webserver started at http://127.9.130.3:33849/ using document root <none> and password file <none>
I20260928 02:39:08.983224 14666 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:08.983270 14666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:08.983407 14666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:08.984537 14666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "56c3590146924bdb8ec9c463865d6bba"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "ad5a12b03b50bff288c14f9e5c79868a"
server_key_iv: "a063f7421c8fd1e42f25fdb342f009da"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.984895 14666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "56c3590146924bdb8ec9c463865d6bba"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "ad5a12b03b50bff288c14f9e5c79868a"
server_key_iv: "a063f7421c8fd1e42f25fdb342f009da"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.986207 14666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:39:08.986936 14680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.987136 14666 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:08.987221 14666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "56c3590146924bdb8ec9c463865d6bba"
format_stamp: "Formatted at 2026-09-28 02:39:08 on dist-test-slave-kkrh"
server_key: "ad5a12b03b50bff288c14f9e5c79868a"
server_key_iv: "a063f7421c8fd1e42f25fdb342f009da"
server_key_version: "encryptionkey@0"
I20260928 02:39:08.987290 14666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:08.990705 14666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:08.990978 14666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:08.991074 14666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:08.991228 14666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:08.991518 14666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:08.991566 14666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.991621 14666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:08.991649 14666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:08.997146 14666 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:41277
I20260928 02:39:08.997190 14793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:41277 every 8 connection(s)
I20260928 02:39:08.997490 14666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:39:09.000298 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.998129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:40951 (local address 127.9.130.62:41463)
0928 02:39:08.998170 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:08.998172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:08.998396 (+ 224us) server_negotiation.cc:407] Connection header received
0928 02:39:08.998880 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:08.998881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:08.998886 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:08.998944 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:09.000079 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:09.000080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:09.000081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:09.000085 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:09.000099 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:09.000103 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:09.000167 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:09.000231 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 02:39:09.000244 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:39:09.000449 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:08.998113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:41463 (local address 127.9.130.3:40951)
0928 02:39:08.998279 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:08.998311 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:08.998771 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:08.998993 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:08.998996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:08.999003 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:08.999942 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:08.999942 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:08.999969 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:08.999996 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:09.000168 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:09.000170 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:09.000173 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:09.000223 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:39:09.000267 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260928 02:39:09.000880 14794 heartbeater.cc:347] Connected to a master server at 127.9.130.62:41463
I20260928 02:39:09.000981 14794 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:09.001137 14794 heartbeater.cc:511] Master 127.9.130.62:41463 requested a full tablet report, sending...
I20260928 02:39:09.001475 14344 ts_manager.cc:194] Registered new tserver with Master: 56c3590146924bdb8ec9c463865d6bba (127.9.130.3:41277)
I20260928 02:39:09.007021 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 14666
I20260928 02:39:09.007148 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:39:09.007326 9736 external_mini_cluster.cc:1428] Setting key 8770389a117a95d8a2eb65b47653aca0
I20260928 02:39:09.008174 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:39:09.010687 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.009526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.9.130.62:41463)
0928 02:39:09.009600 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:09.009602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:09.009712 (+ 110us) server_negotiation.cc:407] Connection header received
0928 02:39:09.009793 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:09.009793 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:09.009799 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:09.009851 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:09.010455 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:09.010457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:09.010457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:09.010462 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:09.010476 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:09.010480 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:09.010536 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:09.010590 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 02:39:09.010605 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:39:09.012537 14344 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:39:09.014031 14344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:39:09.018518 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.016522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:33823 (local address 127.0.0.1:46476)
0928 02:39:09.016719 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:09.016768 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:09.016838 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:09.017065 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:09.017070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:09.017078 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:09.017741 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:09.017741 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:09.017759 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:09.017785 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:09.018158 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:09.018165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:09.018334 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 02:39:09.018398 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:39:09.018415 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":116,"threads_started":1}
I20260928 02:39:09.018550 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.016698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.9.130.1:33823)
0928 02:39:09.016909 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:09.016915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:09.016925 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:39:09.016952 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:09.016954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:09.016964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:09.017037 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:09.017976 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:09.017988 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:09.017989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:09.017995 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:09.018041 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:09.018068 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:09.018143 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:09.018428 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 02:39:09.018449 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260928 02:39:09.018778 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.016987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.9.130.2:39613)
0928 02:39:09.017189 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:09.017193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:09.017248 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:39:09.017344 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:09.017351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:09.017363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:09.017450 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:09.018299 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:09.018306 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:09.018306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:09.018311 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:09.018347 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:09.018372 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:09.018427 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:09.018558 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 02:39:09.018574 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
I20260928 02:39:09.018967 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.016518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:41277 (local address 127.0.0.1:41384)
0928 02:39:09.016719 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:09.016765 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:09.016837 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:09.017585 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:09.017586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:09.017590 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:09.018204 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:09.018205 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:09.018211 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:09.018247 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:09.018828 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:09.018829 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:09.018830 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:39:09.018874 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:39:09.018887 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260928 02:39:09.019156 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.016573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:39613 (local address 127.0.0.1:43174)
0928 02:39:09.017183 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:09.017194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:09.017253 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:09.017464 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:09.017465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:09.017468 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:09.018199 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:09.018199 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:09.018206 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:09.018219 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:09.018455 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:09.018456 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:09.018457 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:39:09.018499 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 02:39:09.018520 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":82,"threads_started":1}
I20260928 02:39:09.019260 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.016812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.9.130.3:41277)
0928 02:39:09.017413 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:09.017418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:09.017429 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:39:09.017455 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:09.017458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:09.017465 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:09.017740 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:09.018661 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:09.018669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:09.018669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:09.018675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:09.018721 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:09.018748 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:09.019097 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:09.019124 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 02:39:09.019154 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":81,"threads_started":1}
I20260928 02:39:09.019311 14466 tablet_service.cc:1467] Processing CreateTablet for tablet 1999262d8b174e6596b6db2002b47cb3 (DEFAULT_TABLE table=test-table [id=ed8bcc159d3e44cea4cae1943b705257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:09.019691 14466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1999262d8b174e6596b6db2002b47cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:09.019877 14597 tablet_service.cc:1467] Processing CreateTablet for tablet 1999262d8b174e6596b6db2002b47cb3 (DEFAULT_TABLE table=test-table [id=ed8bcc159d3e44cea4cae1943b705257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:09.020208 14597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1999262d8b174e6596b6db2002b47cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:09.020355 14728 tablet_service.cc:1467] Processing CreateTablet for tablet 1999262d8b174e6596b6db2002b47cb3 (DEFAULT_TABLE table=test-table [id=ed8bcc159d3e44cea4cae1943b705257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:09.020783 14728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1999262d8b174e6596b6db2002b47cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:09.023119 14808 tablet_bootstrap.cc:492] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1: Bootstrap starting.
I20260928 02:39:09.023479 14810 tablet_bootstrap.cc:492] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba: Bootstrap starting.
I20260928 02:39:09.023479 14809 tablet_bootstrap.cc:492] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16: Bootstrap starting.
I20260928 02:39:09.023980 14808 tablet_bootstrap.cc:654] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:09.024170 14809 tablet_bootstrap.cc:654] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:09.024170 14810 tablet_bootstrap.cc:654] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:09.024336 14808 log.cc:824] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:09.024489 14810 log.cc:824] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:09.024489 14809 log.cc:824] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:09.025229 14809 tablet_bootstrap.cc:492] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16: No bootstrap required, opened a new log
I20260928 02:39:09.025250 14808 tablet_bootstrap.cc:492] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1: No bootstrap required, opened a new log
I20260928 02:39:09.025300 14809 ts_tablet_manager.cc:1397] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:39:09.025300 14808 ts_tablet_manager.cc:1397] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:39:09.025449 14810 tablet_bootstrap.cc:492] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba: No bootstrap required, opened a new log
I20260928 02:39:09.025527 14810 ts_tablet_manager.cc:1397] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:39:09.027576 14810 raft_consensus.cc:348] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f331afb720345ceba101620c028ae16" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39613 } } peers { permanent_uuid: "56c3590146924bdb8ec9c463865d6bba" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41277 } } peers { permanent_uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 33823 } }
I20260928 02:39:09.027736 14810 raft_consensus.cc:374] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:09.027783 14810 raft_consensus.cc:729] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c3590146924bdb8ec9c463865d6bba, State: Initialized, Role: FOLLOWER
I20260928 02:39:09.027948 14810 consensus_queue.cc:260] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f331afb720345ceba101620c028ae16" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39613 } } peers { permanent_uuid: "56c3590146924bdb8ec9c463865d6bba" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41277 } } peers { permanent_uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 33823 } }
I20260928 02:39:09.027904 14809 raft_consensus.cc:348] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f331afb720345ceba101620c028ae16" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39613 } } peers { permanent_uuid: "56c3590146924bdb8ec9c463865d6bba" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41277 } } peers { permanent_uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 33823 } }
I20260928 02:39:09.027904 14808 raft_consensus.cc:348] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f331afb720345ceba101620c028ae16" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39613 } } peers { permanent_uuid: "56c3590146924bdb8ec9c463865d6bba" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41277 } } peers { permanent_uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 33823 } }
I20260928 02:39:09.028053 14808 raft_consensus.cc:374] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:09.028053 14809 raft_consensus.cc:374] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:09.028097 14809 raft_consensus.cc:729] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f331afb720345ceba101620c028ae16, State: Initialized, Role: FOLLOWER
I20260928 02:39:09.028097 14808 raft_consensus.cc:729] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6fa0a0a44c40d68f140c3c3cdacdf1, State: Initialized, Role: FOLLOWER
I20260928 02:39:09.028208 14808 consensus_queue.cc:260] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f331afb720345ceba101620c028ae16" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39613 } } peers { permanent_uuid: "56c3590146924bdb8ec9c463865d6bba" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41277 } } peers { permanent_uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 33823 } }
I20260928 02:39:09.028239 14810 ts_tablet_manager.cc:1428] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 02:39:09.028250 14809 consensus_queue.cc:260] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f331afb720345ceba101620c028ae16" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39613 } } peers { permanent_uuid: "56c3590146924bdb8ec9c463865d6bba" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41277 } } peers { permanent_uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 33823 } }
I20260928 02:39:09.028381 14794 heartbeater.cc:503] Master 127.9.130.62:41463 was elected leader, sending a full tablet report...
I20260928 02:39:09.028467 14808 ts_tablet_manager.cc:1428] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260928 02:39:09.028487 14532 heartbeater.cc:503] Master 127.9.130.62:41463 was elected leader, sending a full tablet report...
I20260928 02:39:09.028734 14809 ts_tablet_manager.cc:1428] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 02:39:09.028770 14663 heartbeater.cc:503] Master 127.9.130.62:41463 was elected leader, sending a full tablet report...
W20260928 02:39:09.133930 14664 tablet.cc:2369] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:39:09.159550 14814 raft_consensus.cc:482] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:39:09.159698 14814 raft_consensus.cc:504] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f331afb720345ceba101620c028ae16" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39613 } } peers { permanent_uuid: "56c3590146924bdb8ec9c463865d6bba" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41277 } } peers { permanent_uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 33823 } }
I20260928 02:39:09.160142 14814 leader_election.cc:288] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f331afb720345ceba101620c028ae16 (127.9.130.2:39613), cb6fa0a0a44c40d68f140c3c3cdacdf1 (127.9.130.1:33823)
I20260928 02:39:09.161597 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.160317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:33823 (local address 127.9.130.3:55801)
0928 02:39:09.160372 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:09.160381 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:09.160420 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:09.160776 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:09.160777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:09.160779 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:09.161237 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:09.161237 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:09.161243 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:09.161255 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:09.161468 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:09.161468 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:09.161469 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:39:09.161521 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:39:09.161535 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 02:39:09.161638 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.160520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:55801 (local address 127.9.130.1:33823)
0928 02:39:09.160642 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:09.160644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:09.160656 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:39:09.160687 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:09.160688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:09.160695 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:09.160765 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:09.161364 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:09.161365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:09.161366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:09.161371 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:09.161390 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:09.161395 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:09.161462 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:09.161532 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:39:09.161548 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:39:09.161701 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.160608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:46903 (local address 127.9.130.2:39613)
0928 02:39:09.160672 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:09.160673 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:09.160682 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:39:09.160708 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:09.160709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:09.160714 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:09.160774 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:09.161386 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:09.161387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:09.161387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:09.161391 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:09.161405 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:09.161410 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:09.161462 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:09.161614 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:39:09.161627 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:39:09.161912 14486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1999262d8b174e6596b6db2002b47cb3" candidate_uuid: "56c3590146924bdb8ec9c463865d6bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1" is_pre_election: true
I20260928 02:39:09.162040 14817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.160477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:39613 (local address 127.9.130.3:46903)
0928 02:39:09.160617 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:09.160627 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:09.160669 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:09.160781 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:09.160782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:09.160795 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:09.161303 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:09.161303 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:09.161308 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:09.161318 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:09.161492 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:09.161509 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:09.161510 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:39:09.161946 (+ 436us) client_negotiation.cc:240] Negotiation successful
0928 02:39:09.161961 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260928 02:39:09.162087 14486 raft_consensus.cc:2454] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56c3590146924bdb8ec9c463865d6bba in term 0.
I20260928 02:39:09.162314 14681 leader_election.cc:302] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c3590146924bdb8ec9c463865d6bba, cb6fa0a0a44c40d68f140c3c3cdacdf1; no voters:
I20260928 02:39:09.162354 14617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1999262d8b174e6596b6db2002b47cb3" candidate_uuid: "56c3590146924bdb8ec9c463865d6bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f331afb720345ceba101620c028ae16" is_pre_election: true
I20260928 02:39:09.162503 14617 raft_consensus.cc:2454] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56c3590146924bdb8ec9c463865d6bba in term 0.
I20260928 02:39:09.162547 14814 raft_consensus.cc:2781] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:39:09.162631 14814 raft_consensus.cc:482] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:39:09.162662 14814 raft_consensus.cc:3037] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:09.163337 14814 raft_consensus.cc:504] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f331afb720345ceba101620c028ae16" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39613 } } peers { permanent_uuid: "56c3590146924bdb8ec9c463865d6bba" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41277 } } peers { permanent_uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 33823 } }
I20260928 02:39:09.163475 14814 leader_election.cc:288] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [CANDIDATE]: Term 1 election: Requested vote from peers 9f331afb720345ceba101620c028ae16 (127.9.130.2:39613), cb6fa0a0a44c40d68f140c3c3cdacdf1 (127.9.130.1:33823)
I20260928 02:39:09.163609 14486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1999262d8b174e6596b6db2002b47cb3" candidate_uuid: "56c3590146924bdb8ec9c463865d6bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1"
I20260928 02:39:09.163631 14617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1999262d8b174e6596b6db2002b47cb3" candidate_uuid: "56c3590146924bdb8ec9c463865d6bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f331afb720345ceba101620c028ae16"
I20260928 02:39:09.163694 14486 raft_consensus.cc:3037] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:09.163700 14617 raft_consensus.cc:3037] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:09.164391 14486 raft_consensus.cc:2454] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c3590146924bdb8ec9c463865d6bba in term 1.
I20260928 02:39:09.164418 14617 raft_consensus.cc:2454] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c3590146924bdb8ec9c463865d6bba in term 1.
I20260928 02:39:09.164572 14681 leader_election.cc:302] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c3590146924bdb8ec9c463865d6bba, cb6fa0a0a44c40d68f140c3c3cdacdf1; no voters:
I20260928 02:39:09.164675 14814 raft_consensus.cc:2781] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:09.164822 14814 raft_consensus.cc:686] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [term 1 LEADER]: Becoming Leader. State: Replica: 56c3590146924bdb8ec9c463865d6bba, State: Running, Role: LEADER
I20260928 02:39:09.164913 14814 consensus_queue.cc:237] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f331afb720345ceba101620c028ae16" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39613 } } peers { permanent_uuid: "56c3590146924bdb8ec9c463865d6bba" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41277 } } peers { permanent_uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 33823 } }
I20260928 02:39:09.165725 14342 catalog_manager.cc:5495] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba reported cstate change: term changed from 0 to 1, leader changed from <none> to 56c3590146924bdb8ec9c463865d6bba (127.9.130.3). New cstate: current_term: 1 leader_uuid: "56c3590146924bdb8ec9c463865d6bba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f331afb720345ceba101620c028ae16" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56c3590146924bdb8ec9c463865d6bba" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 33823 } health_report { overall_health: UNKNOWN } } }
I20260928 02:39:09.201398 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.200373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.9.130.3:41277)
0928 02:39:09.200432 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:09.200433 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:09.200440 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:39:09.200504 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:09.200505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:09.200517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:09.200586 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:09.201177 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:09.201178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:09.201179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:09.201183 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:09.201197 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:09.201201 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:09.201253 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:09.201305 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:39:09.201321 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:39:09.203120 14617 raft_consensus.cc:1261] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16 [term 1 FOLLOWER]: Refusing update from remote peer 56c3590146924bdb8ec9c463865d6bba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:09.203122 14486 raft_consensus.cc:1261] T 1999262d8b174e6596b6db2002b47cb3 P cb6fa0a0a44c40d68f140c3c3cdacdf1 [term 1 FOLLOWER]: Refusing update from remote peer 56c3590146924bdb8ec9c463865d6bba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:09.203449 14814 consensus_queue.cc:1035] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f331afb720345ceba101620c028ae16" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 39613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:09.203606 14818 consensus_queue.cc:1035] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6fa0a0a44c40d68f140c3c3cdacdf1" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 33823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:09.208696 14342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:09.208789 14342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:09.209766 14342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49f245d3ddfb43b5a2531dc9394da43b: Sending DeleteTablet for 3 replicas of tablet 1999262d8b174e6596b6db2002b47cb3
I20260928 02:39:09.210408 14466 tablet_service.cc:1514] Processing DeleteTablet for tablet 1999262d8b174e6596b6db2002b47cb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:09 UTC) from {username='slave'} at 127.0.0.1:46476
I20260928 02:39:09.211002 14597 tablet_service.cc:1514] Processing DeleteTablet for tablet 1999262d8b174e6596b6db2002b47cb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:09 UTC) from {username='slave'} at 127.0.0.1:43174
I20260928 02:39:09.211167 14728 tablet_service.cc:1514] Processing DeleteTablet for tablet 1999262d8b174e6596b6db2002b47cb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:09 UTC) from {username='slave'} at 127.0.0.1:41384
I20260928 02:39:09.211304 14833 tablet_replica.cc:330] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16: stopping tablet replica
I20260928 02:39:09.211515 14833 raft_consensus.cc:2229] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:09.211891 14833 raft_consensus.cc:2258] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:09.212096 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 14404
I20260928 02:39:09.212709 14833 ts_tablet_manager.cc:1905] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:09.212939 14834 tablet_replica.cc:330] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba: stopping tablet replica
I20260928 02:39:09.213033 14834 raft_consensus.cc:2229] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:39:09.213166 14834 raft_consensus.cc:2258] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:09.213536 14834 ts_tablet_manager.cc:1905] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:09.214985 14833 ts_tablet_manager.cc:1918] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:09.215111 14833 log.cc:1196] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/1999262d8b174e6596b6db2002b47cb3
I20260928 02:39:09.215119 14834 ts_tablet_manager.cc:1918] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:09.215173 14834 log.cc:1196] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/1999262d8b174e6596b6db2002b47cb3
I20260928 02:39:09.215474 14833 ts_tablet_manager.cc:1939] T 1999262d8b174e6596b6db2002b47cb3 P 9f331afb720345ceba101620c028ae16: Deleting consensus metadata
I20260928 02:39:09.215474 14834 ts_tablet_manager.cc:1939] T 1999262d8b174e6596b6db2002b47cb3 P 56c3590146924bdb8ec9c463865d6bba: Deleting consensus metadata
I20260928 02:39:09.215894 14331 catalog_manager.cc:4841] TS 56c3590146924bdb8ec9c463865d6bba (127.9.130.3:41277): tablet 1999262d8b174e6596b6db2002b47cb3 (table test-table [id=ed8bcc159d3e44cea4cae1943b705257]) successfully deleted
I20260928 02:39:09.215894 14330 catalog_manager.cc:4841] TS 9f331afb720345ceba101620c028ae16 (127.9.130.2:39613): tablet 1999262d8b174e6596b6db2002b47cb3 (table test-table [id=ed8bcc159d3e44cea4cae1943b705257]) successfully deleted
W20260928 02:39:09.221091 14328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:33823 (error 108)
I20260928 02:39:09.221351 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 14535
I20260928 02:39:09.222002 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.221463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:33823 (local address 127.0.0.1:46480)
0928 02:39:09.221919 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:09.221959 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437}
W20260928 02:39:09.223057 14328 catalog_manager.cc:4584] TS cb6fa0a0a44c40d68f140c3c3cdacdf1 (127.9.130.1:33823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1999262d8b174e6596b6db2002b47cb3: Network error: Client connection negotiation failed: client connection to 127.9.130.1:33823: connect: Connection refused (error 111)
I20260928 02:39:09.226351 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 14666
I20260928 02:39:09.230796 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 14313
2026-09-28T02:39:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (723 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 02:39:09.249715 9736 mini_oidc.cc:152] Starting JWKS server
I20260928 02:39:09.250676 9736 webserver.cc:460] Webserver started at https://127.0.0.1:45833/ using document root <none> and password file <none>
I20260928 02:39:09.250751 9736 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:39:09.250892 9736 webserver.cc:460] Webserver started at http://127.0.0.1:44557/ using document root <none> and password file <none>
2026-09-28T02:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:09Z Disabled control of system clock
I20260928 02:39:09.258428 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:39079
--webserver_interface=127.9.130.62
--webserver_port=0
--builtin_ntp_servers=127.9.130.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:39079
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45833/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/testchainca.cert with env {}
W20260928 02:39:09.337260 14841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:09.337466 14841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:09.337488 14841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:09.338605 14841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:09.338647 14841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:09.338660 14841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:09.338673 14841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:09.338685 14841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:39:09.338697 14841 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:39:09.338708 14841 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45833/jwks.json
I20260928 02:39:09.340262 14841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:39079
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:39079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45833/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:09.340459 14841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:09.340634 14841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:09.345239 14847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:09.345250 14846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:09.345323 14841 server_base.cc:1034] running on GCE node
W20260928 02:39:09.345247 14849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:09.345710 14841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:09.346007 14841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:09.347159 14841 hybrid_clock.cc:648] HybridClock initialized: now 1790563149347157 us; error 41 us; skew 500 ppm
I20260928 02:39:09.348531 14841 webserver.cc:460] Webserver started at http://127.9.130.62:33555/ using document root <none> and password file <none>
I20260928 02:39:09.348767 14841 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:09.348834 14841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:09.348987 14841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:09.350018 14841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "efd0cb864eb64eae8f03dcb9d8e6a322"
format_stamp: "Formatted at 2026-09-28 02:39:09 on dist-test-slave-kkrh"
server_key: "628750fe0f5545344f4ce782cb6a63ae"
server_key_iv: "7340e89e3b8af143c73d635ae07f9f65"
server_key_version: "encryptionkey@0"
I20260928 02:39:09.350414 14841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "efd0cb864eb64eae8f03dcb9d8e6a322"
format_stamp: "Formatted at 2026-09-28 02:39:09 on dist-test-slave-kkrh"
server_key: "628750fe0f5545344f4ce782cb6a63ae"
server_key_iv: "7340e89e3b8af143c73d635ae07f9f65"
server_key_version: "encryptionkey@0"
I20260928 02:39:09.351725 14841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:39:09.352546 14855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:09.352711 14841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:09.352803 14841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "efd0cb864eb64eae8f03dcb9d8e6a322"
format_stamp: "Formatted at 2026-09-28 02:39:09 on dist-test-slave-kkrh"
server_key: "628750fe0f5545344f4ce782cb6a63ae"
server_key_iv: "7340e89e3b8af143c73d635ae07f9f65"
server_key_version: "encryptionkey@0"
I20260928 02:39:09.352877 14841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:09.380230 14841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:09.380596 14841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:09.380764 14841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:09.384639 14841 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:39079
I20260928 02:39:09.384675 14907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:39079 every 8 connection(s)
I20260928 02:39:09.385038 14841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:09.386003 14908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:09.388289 14908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322: Bootstrap starting.
I20260928 02:39:09.388919 14908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:09.389213 14908 log.cc:824] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:09.389914 14908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322: No bootstrap required, opened a new log
I20260928 02:39:09.391682 14908 raft_consensus.cc:348] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd0cb864eb64eae8f03dcb9d8e6a322" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39079 } }
I20260928 02:39:09.391876 14908 raft_consensus.cc:374] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:09.391903 14908 raft_consensus.cc:729] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd0cb864eb64eae8f03dcb9d8e6a322, State: Initialized, Role: FOLLOWER
I20260928 02:39:09.392068 14908 consensus_queue.cc:260] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd0cb864eb64eae8f03dcb9d8e6a322" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39079 } }
I20260928 02:39:09.392150 14908 raft_consensus.cc:388] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:09.392190 14908 raft_consensus.cc:482] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:09.392241 14908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:09.392549 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 14841
I20260928 02:39:09.392669 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:09.392848 9736 external_mini_cluster.cc:1428] Setting key 48ad7ad4257f6f1e6566cda8e1404984
I20260928 02:39:09.392856 14908 raft_consensus.cc:504] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd0cb864eb64eae8f03dcb9d8e6a322" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39079 } }
I20260928 02:39:09.392978 14908 leader_election.cc:302] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efd0cb864eb64eae8f03dcb9d8e6a322; no voters:
I20260928 02:39:09.393139 14908 leader_election.cc:288] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:09.393179 14911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:09.393318 14911 raft_consensus.cc:686] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [term 1 LEADER]: Becoming Leader. State: Replica: efd0cb864eb64eae8f03dcb9d8e6a322, State: Running, Role: LEADER
I20260928 02:39:09.393397 14908 sys_catalog.cc:564] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:09.393405 14911 consensus_queue.cc:237] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd0cb864eb64eae8f03dcb9d8e6a322" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39079 } }
I20260928 02:39:09.393884 14914 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efd0cb864eb64eae8f03dcb9d8e6a322" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd0cb864eb64eae8f03dcb9d8e6a322" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39079 } } }
I20260928 02:39:09.393998 14914 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:09.394168 14912 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efd0cb864eb64eae8f03dcb9d8e6a322. Latest consensus state: current_term: 1 leader_uuid: "efd0cb864eb64eae8f03dcb9d8e6a322" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd0cb864eb64eae8f03dcb9d8e6a322" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39079 } } }
I20260928 02:39:09.394240 14912 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:09.394587 14922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:09.395146 14922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:09.396617 14922 catalog_manager.cc:1347] Generated new cluster ID: 1ed54ae8e05f4993bfbcca2ad1bb31db
I20260928 02:39:09.396667 14922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:39:09.400015 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.393403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.9.130.62:39079)
0928 02:39:09.394018 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:09.394028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:09.394060 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:39:09.394185 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:09.394193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:09.394262 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:09.394398 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:09.394683 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.396325 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.397435 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.398072 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.398192 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:09.399082 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:09.399087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:09.399087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:09.399304 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:09.399344 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:09.399358 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:09.399417 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:09.399692 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 02:39:09.399771 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":66,"threads_started":1}
I20260928 02:39:09.400820 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:39:09.404748 14922 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:09.405228 14922 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:09.411454 14922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efd0cb864eb64eae8f03dcb9d8e6a322: Generated new TSK 0
I20260928 02:39:09.411652 14922 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:39:09.432732 14938 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 02:39:09.432837 14938 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:39:09.433090 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.417967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.9.130.62:39079)
0928 02:39:09.418013 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:09.418016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:09.418164 (+ 148us) server_negotiation.cc:407] Connection header received
0928 02:39:09.418267 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:09.418267 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:09.418288 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:09.418347 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:39:09.418517 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.418986 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.420135 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.420312 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.420364 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:09.420575 (+ 211us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:39:09.432864 (+ 12289us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:39:09.433025 (+ 161us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 02:39:09.433151 14915 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 02:39:09.433421 14937 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:09.417910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39079 (local address 127.0.0.1:50348)
0928 02:39:09.418091 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:09.418103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:09.418176 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:09.418366 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:09.418368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:09.418370 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:39:09.418448 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:09.418451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.419071 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.419072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:09.420059 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:09.420062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.420381 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.420382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:09.420459 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:09.420478 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:39:09.433015 (+ 12537us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:39:09.433127 (+ 112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.130.62:39079: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.130.62:39079: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 02:39:09.433961 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 14841
2026-09-28T02:39:09Z chronyd exiting
I20260928 02:39:09.469913 9736 test_util.cc:175] -----------------------------------------------
I20260928 02:39:09.469998 9736 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563127283700-9736-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (221 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 02:39:09.470786 9736 mini_oidc.cc:152] Starting JWKS server
I20260928 02:39:09.471490 9736 webserver.cc:460] Webserver started at https://127.0.0.1:38691/ using document root <none> and password file <none>
I20260928 02:39:09.471560 9736 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:39:09.471750 9736 webserver.cc:460] Webserver started at http://127.0.0.1:38413/ using document root <none> and password file <none>
2026-09-28T02:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:09Z Disabled control of system clock
I20260928 02:39:09.479264 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:39583
--webserver_interface=127.9.130.62
--webserver_port=0
--builtin_ntp_servers=127.9.130.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:39583
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38691/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/testchainca.cert with env {}
W20260928 02:39:09.558060 14944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:09.558297 14944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:09.558332 14944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:09.559484 14944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:09.559544 14944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:09.559564 14944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:09.559593 14944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:09.559620 14944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:39:09.559643 14944 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:39:09.559667 14944 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38691/jwks.json
I20260928 02:39:09.561137 14944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:39583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:39583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38691/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:09.561411 14944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:09.561626 14944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:39:09.566390 14944 server_base.cc:1034] running on GCE node
W20260928 02:39:09.566521 14950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:09.566496 14952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:09.566392 14949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:09.566954 14944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:09.567232 14944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:09.568391 14944 hybrid_clock.cc:648] HybridClock initialized: now 1790563149568381 us; error 34 us; skew 500 ppm
I20260928 02:39:09.569741 14944 webserver.cc:460] Webserver started at http://127.9.130.62:45015/ using document root <none> and password file <none>
I20260928 02:39:09.569983 14944 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:09.570048 14944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:09.570197 14944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:09.571252 14944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "0114d3aca94d4439afbed66a60e6d736"
format_stamp: "Formatted at 2026-09-28 02:39:09 on dist-test-slave-kkrh"
server_key: "6d9548f7d2c1705a6e24a5d920fabcad"
server_key_iv: "f8b86cdf58b745e2384c5ac5053aa12f"
server_key_version: "encryptionkey@0"
I20260928 02:39:09.571606 14944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "0114d3aca94d4439afbed66a60e6d736"
format_stamp: "Formatted at 2026-09-28 02:39:09 on dist-test-slave-kkrh"
server_key: "6d9548f7d2c1705a6e24a5d920fabcad"
server_key_iv: "f8b86cdf58b745e2384c5ac5053aa12f"
server_key_version: "encryptionkey@0"
I20260928 02:39:09.572957 14944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:39:09.573657 14958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:09.573825 14944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:09.573916 14944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "0114d3aca94d4439afbed66a60e6d736"
format_stamp: "Formatted at 2026-09-28 02:39:09 on dist-test-slave-kkrh"
server_key: "6d9548f7d2c1705a6e24a5d920fabcad"
server_key_iv: "f8b86cdf58b745e2384c5ac5053aa12f"
server_key_version: "encryptionkey@0"
I20260928 02:39:09.573997 14944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:09.596669 14944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:09.596978 14944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:09.597131 14944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:09.600879 14944 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:39583
I20260928 02:39:09.600915 15010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:39583 every 8 connection(s)
I20260928 02:39:09.601275 14944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:09.602216 15011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:09.603396 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 14944
I20260928 02:39:09.603470 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:09.603645 9736 external_mini_cluster.cc:1428] Setting key 47bf62ddf8eb5a70440e8ff30ad09687
I20260928 02:39:09.604724 15011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736: Bootstrap starting.
I20260928 02:39:09.605280 15011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:09.605556 15011 log.cc:824] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:09.606190 15011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736: No bootstrap required, opened a new log
I20260928 02:39:09.608553 15011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0114d3aca94d4439afbed66a60e6d736" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39583 } }
I20260928 02:39:09.608741 15011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:09.608791 15011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0114d3aca94d4439afbed66a60e6d736, State: Initialized, Role: FOLLOWER
I20260928 02:39:09.608917 15011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0114d3aca94d4439afbed66a60e6d736" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39583 } }
I20260928 02:39:09.609002 15011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:09.609041 15011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:09.609084 15011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:09.609349 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.604153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.9.130.62:39583)
0928 02:39:09.604356 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:09.604365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:09.604396 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:39:09.604807 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:09.604832 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:09.604912 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:09.605051 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:09.605284 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.606244 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.607372 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.607853 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.607961 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:09.608686 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:09.608689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:09.608689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:09.608913 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:09.608946 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:09.608957 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:09.609029 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:09.609108 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:39:09.609188 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260928 02:39:09.609771 15011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0114d3aca94d4439afbed66a60e6d736" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39583 } }
I20260928 02:39:09.609871 15011 leader_election.cc:302] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0114d3aca94d4439afbed66a60e6d736; no voters:
I20260928 02:39:09.610025 15011 leader_election.cc:288] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:09.610049 15016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:09.610157 15016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [term 1 LEADER]: Becoming Leader. State: Replica: 0114d3aca94d4439afbed66a60e6d736, State: Running, Role: LEADER
I20260928 02:39:09.610229 15016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0114d3aca94d4439afbed66a60e6d736" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39583 } }
I20260928 02:39:09.610241 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:39:09.610263 15011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:09.610525 15016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0114d3aca94d4439afbed66a60e6d736" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0114d3aca94d4439afbed66a60e6d736" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39583 } } }
I20260928 02:39:09.610627 15016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:09.610750 15016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0114d3aca94d4439afbed66a60e6d736. Latest consensus state: current_term: 1 leader_uuid: "0114d3aca94d4439afbed66a60e6d736" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0114d3aca94d4439afbed66a60e6d736" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39583 } } }
I20260928 02:39:09.610822 15016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:09.611912 15021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:09.612455 15021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:09.613977 15021 catalog_manager.cc:1347] Generated new cluster ID: e42d0953364145ec98da7d5c531f4ab3
I20260928 02:39:09.614033 15021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:39:09.626894 15021 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:09.627700 15021 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:09.647598 15021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0114d3aca94d4439afbed66a60e6d736: Generated new TSK 0
I20260928 02:39:09.647809 15021 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:39:09.672214 15041 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 02:39:09.672295 15041 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:39:09.672572 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.657659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44120 (local address 127.9.130.62:39583)
0928 02:39:09.657723 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:09.657725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:09.657841 (+ 116us) server_negotiation.cc:407] Connection header received
0928 02:39:09.657932 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:09.657932 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:09.657943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:09.658004 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:39:09.658155 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.658641 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.659910 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.660096 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.660148 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:09.660338 (+ 190us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:39:09.672408 (+ 12070us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38691/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:39:09.672523 (+ 115us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44120: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38691/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 02:39:09.672644 15014 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44120: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38691/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 02:39:09.672739 15040 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:09.657608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39583 (local address 127.0.0.1:44120)
0928 02:39:09.657766 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:09.657780 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:09.657853 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:09.658023 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:09.658024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:09.658026 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:39:09.658089 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:09.658091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.658714 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.658716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:09.659792 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:09.659795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.660165 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.660166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:09.660232 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:09.660237 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:39:09.672511 (+ 12274us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38691/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:39:09.672631 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.130.62:39583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38691/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260928 02:39:09.673102 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 14944
2026-09-28T02:39:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (240 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 02:39:09.710933 9736 mini_oidc.cc:152] Starting JWKS server
I20260928 02:39:09.711670 9736 webserver.cc:460] Webserver started at https://127.0.0.1:40783/ using document root <none> and password file <none>
I20260928 02:39:09.711742 9736 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:39:09.711982 9736 webserver.cc:460] Webserver started at http://127.0.0.1:41637/ using document root <none> and password file <none>
2026-09-28T02:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:09Z Disabled control of system clock
I20260928 02:39:09.719748 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:42349
--webserver_interface=127.9.130.62
--webserver_port=0
--builtin_ntp_servers=127.9.130.20:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:42349
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40783/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:39:09.800107 15047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:09.800293 15047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:09.800314 15047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:09.801445 15047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:09.801486 15047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:09.801497 15047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:09.801510 15047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:09.801563 15047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:39:09.801587 15047 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:39:09.801614 15047 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40783/jwks.json
I20260928 02:39:09.802943 15047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:42349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:42349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40783/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:09.803177 15047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:09.803440 15047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:09.808147 15052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:09.808176 15055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:09.808176 15053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:09.808215 15047 server_base.cc:1034] running on GCE node
I20260928 02:39:09.808538 15047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:09.808768 15047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:09.809898 15047 hybrid_clock.cc:648] HybridClock initialized: now 1790563149809888 us; error 27 us; skew 500 ppm
I20260928 02:39:09.811092 15047 webserver.cc:460] Webserver started at http://127.9.130.62:34295/ using document root <none> and password file <none>
I20260928 02:39:09.811311 15047 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:09.811355 15047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:09.811511 15047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:09.812608 15047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "13e89eebd1b84ac5a196319241ac1010"
format_stamp: "Formatted at 2026-09-28 02:39:09 on dist-test-slave-kkrh"
server_key: "c58c311b02f712ff508ccb8d47f6b46f"
server_key_iv: "17b2c745e5b5e04d9dbd08716a75b8e2"
server_key_version: "encryptionkey@0"
I20260928 02:39:09.812951 15047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "13e89eebd1b84ac5a196319241ac1010"
format_stamp: "Formatted at 2026-09-28 02:39:09 on dist-test-slave-kkrh"
server_key: "c58c311b02f712ff508ccb8d47f6b46f"
server_key_iv: "17b2c745e5b5e04d9dbd08716a75b8e2"
server_key_version: "encryptionkey@0"
I20260928 02:39:09.814189 15047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:39:09.814888 15061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:09.815040 15047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:09.815138 15047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "13e89eebd1b84ac5a196319241ac1010"
format_stamp: "Formatted at 2026-09-28 02:39:09 on dist-test-slave-kkrh"
server_key: "c58c311b02f712ff508ccb8d47f6b46f"
server_key_iv: "17b2c745e5b5e04d9dbd08716a75b8e2"
server_key_version: "encryptionkey@0"
I20260928 02:39:09.815214 15047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:09.831288 15047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:09.831584 15047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:09.831736 15047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:09.835359 15047 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:42349
I20260928 02:39:09.835399 15113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:42349 every 8 connection(s)
I20260928 02:39:09.835734 15047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:09.836386 15114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:09.839077 15114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010: Bootstrap starting.
I20260928 02:39:09.839639 15114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:09.839941 15114 log.cc:824] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:09.840569 15114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010: No bootstrap required, opened a new log
I20260928 02:39:09.842098 15114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e89eebd1b84ac5a196319241ac1010" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42349 } }
I20260928 02:39:09.842229 15114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:09.842252 15114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e89eebd1b84ac5a196319241ac1010, State: Initialized, Role: FOLLOWER
I20260928 02:39:09.842427 15114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e89eebd1b84ac5a196319241ac1010" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42349 } }
I20260928 02:39:09.842490 15114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:09.842530 15114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:09.842576 15114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:09.843070 15114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e89eebd1b84ac5a196319241ac1010" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42349 } }
I20260928 02:39:09.843176 15114 leader_election.cc:302] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13e89eebd1b84ac5a196319241ac1010; no voters:
I20260928 02:39:09.843353 15114 leader_election.cc:288] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:09.843412 15117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:09.843554 15117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [term 1 LEADER]: Becoming Leader. State: Replica: 13e89eebd1b84ac5a196319241ac1010, State: Running, Role: LEADER
I20260928 02:39:09.843590 15114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:09.843701 15117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e89eebd1b84ac5a196319241ac1010" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42349 } }
I20260928 02:39:09.843876 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 15047
I20260928 02:39:09.843966 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:09.844213 9736 external_mini_cluster.cc:1428] Setting key efa61b3128dd38d57aa6e1a76ddc9e45
I20260928 02:39:09.844861 15118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13e89eebd1b84ac5a196319241ac1010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e89eebd1b84ac5a196319241ac1010" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42349 } } }
I20260928 02:39:09.844949 15118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:09.845234 15119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13e89eebd1b84ac5a196319241ac1010. Latest consensus state: current_term: 1 leader_uuid: "13e89eebd1b84ac5a196319241ac1010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e89eebd1b84ac5a196319241ac1010" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42349 } } }
I20260928 02:39:09.845427 15119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010 [sys.catalog]: This master's current role is: LEADER
W20260928 02:39:09.845665 15135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:39:09.845726 15135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:39:09.845791 15133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:09.845968 15133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:09.847230 15133 catalog_manager.cc:1347] Generated new cluster ID: 01099003b2774b5080b34ca0745d08bf
I20260928 02:39:09.847285 15133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:39:09.850234 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.844833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.9.130.62:42349)
0928 02:39:09.845005 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:09.845014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:09.845047 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:39:09.845160 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:09.845168 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:09.845245 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:09.845649 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:09.845750 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.847087 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.848172 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.848808 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.848923 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:09.849550 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:09.849553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:09.849554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:09.849771 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:09.849812 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:09.849827 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:09.849892 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:09.849962 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:39:09.850037 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260928 02:39:09.851277 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:39:09.867800 15133 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:09.868409 15133 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:09.877146 15133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13e89eebd1b84ac5a196319241ac1010: Generated new TSK 0
I20260928 02:39:09.877368 15133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:39:09.890111 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:09.886539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.9.130.62:42349)
0928 02:39:09.886601 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:09.886603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:09.886687 (+ 84us) server_negotiation.cc:407] Connection header received
0928 02:39:09.886755 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:09.886756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:09.886767 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:09.886845 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:39:09.886943 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.887696 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.888502 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.888809 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.888879 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:09.888999 (+ 120us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:39:09.889905 (+ 906us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40783/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:39:09.890033 (+ 128us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40783/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 02:39:09.890168 15129 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40783/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 02:39:09.892221 15143 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:09.886357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:42349 (local address 127.0.0.1:58772)
0928 02:39:09.886607 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:09.886618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:09.886701 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:09.886849 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:09.886850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:09.886851 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:39:09.886900 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:09.886902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.887791 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.887792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:09.888425 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:09.888428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:09.888880 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:09.888880 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:09.888917 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:09.888922 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:39:09.890008 (+ 1086us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40783/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:39:09.892120 (+ 2112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.130.62:42349: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40783/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":1933,"thread_start_us":201,"threads_started":1}
I20260928 02:39:09.892498 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 15047
E20260928 02:39:09.901530 15144 webserver.cc:533] Webserver: sslize failed:
E20260928 02:39:09.901598 15144 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T02:39:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T02:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:09Z Disabled control of system clock
I20260928 02:39:09.937158 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:39757
--webserver_interface=127.9.130.62
--webserver_port=0
--builtin_ntp_servers=127.9.130.20:42237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:39757
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/testchainca.cert with env {}
W20260928 02:39:10.015511 15148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:10.015717 15148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:10.015749 15148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:10.016964 15148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:10.017010 15148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:10.017022 15148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:10.017035 15148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:10.017047 15148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:39:10.017066 15148 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:39:10.017077 15148 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 02:39:10.018550 15148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:42237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:39757
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:39757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:10.018746 15148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:10.018931 15148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:10.023923 15156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:10.023841 15154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:10.023845 15153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:10.023891 15148 server_base.cc:1034] running on GCE node
I20260928 02:39:10.024356 15148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:10.024679 15148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:10.025862 15148 hybrid_clock.cc:648] HybridClock initialized: now 1790563150025849 us; error 34 us; skew 500 ppm
I20260928 02:39:10.027163 15148 webserver.cc:460] Webserver started at http://127.9.130.62:35463/ using document root <none> and password file <none>
I20260928 02:39:10.027415 15148 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:10.027484 15148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:10.027626 15148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:10.028766 15148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "7b8089abc6ac4746931d2882bf2bc697"
format_stamp: "Formatted at 2026-09-28 02:39:10 on dist-test-slave-kkrh"
server_key: "ace032b9dc1d33f515e1547d85d9d06f"
server_key_iv: "99b0e30c1774944ed7161af0cbd8e2d3"
server_key_version: "encryptionkey@0"
I20260928 02:39:10.029160 15148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "7b8089abc6ac4746931d2882bf2bc697"
format_stamp: "Formatted at 2026-09-28 02:39:10 on dist-test-slave-kkrh"
server_key: "ace032b9dc1d33f515e1547d85d9d06f"
server_key_iv: "99b0e30c1774944ed7161af0cbd8e2d3"
server_key_version: "encryptionkey@0"
I20260928 02:39:10.030517 15148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:10.031334 15162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:10.031509 15148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:10.031600 15148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "7b8089abc6ac4746931d2882bf2bc697"
format_stamp: "Formatted at 2026-09-28 02:39:10 on dist-test-slave-kkrh"
server_key: "ace032b9dc1d33f515e1547d85d9d06f"
server_key_iv: "99b0e30c1774944ed7161af0cbd8e2d3"
server_key_version: "encryptionkey@0"
I20260928 02:39:10.031708 15148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:10.046110 15148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:10.046407 15148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:10.046552 15148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:10.050287 15148 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:39757
I20260928 02:39:10.050329 15214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:39757 every 8 connection(s)
I20260928 02:39:10.050704 15148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:10.051096 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 15148
I20260928 02:39:10.051225 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:10.051468 9736 external_mini_cluster.cc:1428] Setting key 86ca1893f63719df3fcb7e57aff3fa45
I20260928 02:39:10.051891 15215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:10.055382 15215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697: Bootstrap starting.
I20260928 02:39:10.056152 15215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:10.056476 15215 log.cc:824] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:10.057228 15215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697: No bootstrap required, opened a new log
I20260928 02:39:10.057700 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:10.052044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.9.130.62:39757)
0928 02:39:10.052276 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:10.052285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:10.052319 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:39:10.053042 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:10.053051 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:10.053136 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:10.053295 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:10.053471 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:10.054794 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:10.055597 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:10.056246 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:10.056407 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:10.057034 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:10.057039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:10.057039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:10.057217 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:10.057250 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:10.057261 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:10.057337 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:10.057453 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 02:39:10.057533 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
I20260928 02:39:10.058585 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:39:10.059628 15215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8089abc6ac4746931d2882bf2bc697" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39757 } }
I20260928 02:39:10.059798 15215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:10.059868 15215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8089abc6ac4746931d2882bf2bc697, State: Initialized, Role: FOLLOWER
I20260928 02:39:10.059998 15215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8089abc6ac4746931d2882bf2bc697" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39757 } }
I20260928 02:39:10.060081 15215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:10.060127 15215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:10.060179 15215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:10.060753 15215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8089abc6ac4746931d2882bf2bc697" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39757 } }
I20260928 02:39:10.060876 15215 leader_election.cc:302] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b8089abc6ac4746931d2882bf2bc697; no voters:
I20260928 02:39:10.061038 15215 leader_election.cc:288] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:10.061084 15220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:10.061239 15220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8089abc6ac4746931d2882bf2bc697, State: Running, Role: LEADER
I20260928 02:39:10.061295 15215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:10.061321 15220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8089abc6ac4746931d2882bf2bc697" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39757 } }
I20260928 02:39:10.061655 15221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b8089abc6ac4746931d2882bf2bc697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8089abc6ac4746931d2882bf2bc697" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39757 } } }
I20260928 02:39:10.061668 15222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b8089abc6ac4746931d2882bf2bc697. Latest consensus state: current_term: 1 leader_uuid: "7b8089abc6ac4746931d2882bf2bc697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8089abc6ac4746931d2882bf2bc697" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39757 } } }
I20260928 02:39:10.061777 15221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:10.061800 15222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:10.062065 15228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:10.062511 15228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:10.064033 15228 catalog_manager.cc:1347] Generated new cluster ID: 6955dd1fd5a14736961e07c9f99e8092
I20260928 02:39:10.064087 15228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:39:10.070047 15228 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:10.070524 15228 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:10.074862 15228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b8089abc6ac4746931d2882bf2bc697: Generated new TSK 0
I20260928 02:39:10.075014 15228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:39:10.502184 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:10.081942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37966 (local address 127.9.130.62:39757)
0928 02:39:10.081998 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:10.082000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:10.082089 (+ 89us) server_negotiation.cc:407] Connection header received
0928 02:39:10.082140 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:10.082141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:10.082153 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:10.082230 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:39:10.082329 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:10.083097 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:10.083811 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:10.084145 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:10.084215 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:10.084355 (+ 140us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:39:10.501938 (+417583us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:39:10.502124 (+ 186us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37966: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 02:39:10.502267 15218 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37966: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 02:39:10.502372 15244 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:10.081863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39757 (local address 127.0.0.1:37966)
0928 02:39:10.082021 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:10.082031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:10.082090 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:10.082229 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:10.082230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:10.082232 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:39:10.082276 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:10.082278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:10.083180 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:10.083181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:10.083742 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:10.083744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:10.084216 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:10.084217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:10.084265 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:10.084270 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:39:10.502117 (+417847us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:39:10.502250 (+ 133us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.130.62:39757: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260928 02:39:10.502681 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 15148
2026-09-28T02:39:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (590 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (84 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:39:10 dist-test-slave-kkrh krb5kdc[15256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:39:10 dist-test-slave-kkrh krb5kdc[15256](info): set up 2 sockets
Sep 28 02:39:10 dist-test-slave-kkrh krb5kdc[15256](info): commencing operation
krb5kdc: starting...
W20260928 02:39:11.708081 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:39:11 dist-test-slave-kkrh krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:39:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:39:11.823099 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:41985
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:41985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:11.901980 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:11.902176 15269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:11.902197 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:11.903688 15269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:11.903735 15269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:11.903760 15269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:11.903793 15269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:11.903833 15269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:11.905308 15269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:41985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:41985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:11.905490 15269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:11.905648 15269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:11.910264 15275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:11.910301 15274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:11.910425 15269 server_base.cc:1034] running on GCE node
W20260928 02:39:11.910279 15277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:11.910729 15269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:11.911032 15269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:11.912179 15269 hybrid_clock.cc:648] HybridClock initialized: now 1790563151912161 us; error 36 us; skew 500 ppm
Sep 28 02:39:11 dist-test-slave-kkrh krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563151, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:11.917923 15269 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:39:11.918361 15269 webserver.cc:460] Webserver started at http://127.9.130.62:45551/ using document root <none> and password file <none>
I20260928 02:39:11.918586 15269 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:11.918648 15269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:11.918785 15269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:11.919793 15269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "fa37144c0d75411c8d63d9ff28d61c45"
format_stamp: "Formatted at 2026-09-28 02:39:11 on dist-test-slave-kkrh"
server_key: "e8bb0b0724670a1be14d822964de670b"
server_key_iv: "78b7f0a99d0eb034beeac60d016b6a4b"
server_key_version: "encryptionkey@0"
I20260928 02:39:11.920200 15269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "fa37144c0d75411c8d63d9ff28d61c45"
format_stamp: "Formatted at 2026-09-28 02:39:11 on dist-test-slave-kkrh"
server_key: "e8bb0b0724670a1be14d822964de670b"
server_key_iv: "78b7f0a99d0eb034beeac60d016b6a4b"
server_key_version: "encryptionkey@0"
I20260928 02:39:11.921489 15269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:11.922317 15284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:11.922542 15269 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:11.922657 15269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "fa37144c0d75411c8d63d9ff28d61c45"
format_stamp: "Formatted at 2026-09-28 02:39:11 on dist-test-slave-kkrh"
server_key: "e8bb0b0724670a1be14d822964de670b"
server_key_iv: "78b7f0a99d0eb034beeac60d016b6a4b"
server_key_version: "encryptionkey@0"
I20260928 02:39:11.922776 15269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:11.935606 15269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:11.937764 15269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:11.937988 15269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:11.942183 15269 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:41985
I20260928 02:39:11.942209 15336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:41985 every 8 connection(s)
I20260928 02:39:11.942574 15269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:11.943396 15337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:11.945709 15337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45: Bootstrap starting.
I20260928 02:39:11.946259 15337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:11.946532 15337 log.cc:824] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:11.947235 15337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45: No bootstrap required, opened a new log
I20260928 02:39:11.947535 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 15269
I20260928 02:39:11.947649 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:11.947978 9736 external_mini_cluster.cc:1428] Setting key c291212d0e4d2031cb67a8034ef44d21
I20260928 02:39:11.949272 15337 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa37144c0d75411c8d63d9ff28d61c45" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41985 } }
I20260928 02:39:11.949429 15337 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:11.949469 15337 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa37144c0d75411c8d63d9ff28d61c45, State: Initialized, Role: FOLLOWER
I20260928 02:39:11.949600 15337 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa37144c0d75411c8d63d9ff28d61c45" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41985 } }
I20260928 02:39:11.949683 15337 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:11.949719 15337 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:11.949759 15337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:11.950709 15337 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa37144c0d75411c8d63d9ff28d61c45" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41985 } }
I20260928 02:39:11.950791 15337 leader_election.cc:302] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa37144c0d75411c8d63d9ff28d61c45; no voters:
I20260928 02:39:11.950960 15337 leader_election.cc:288] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:11.950999 15342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:11.951131 15342 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [term 1 LEADER]: Becoming Leader. State: Replica: fa37144c0d75411c8d63d9ff28d61c45, State: Running, Role: LEADER
I20260928 02:39:11.951200 15337 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:11.951208 15342 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa37144c0d75411c8d63d9ff28d61c45" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41985 } }
I20260928 02:39:11.951552 15343 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa37144c0d75411c8d63d9ff28d61c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa37144c0d75411c8d63d9ff28d61c45" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41985 } } }
I20260928 02:39:11.951643 15343 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:11.951545 15342 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa37144c0d75411c8d63d9ff28d61c45. Latest consensus state: current_term: 1 leader_uuid: "fa37144c0d75411c8d63d9ff28d61c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa37144c0d75411c8d63d9ff28d61c45" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41985 } } }
I20260928 02:39:11.951835 15342 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:11.951968 15348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:11.952401 15348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:11.953822 15348 catalog_manager.cc:1347] Generated new cluster ID: 4b14d204264d4eb58f32811177bc969f
I20260928 02:39:11.953869 15348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:39:11 dist-test-slave-kkrh krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:11.959302 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:11.948499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.9.130.62:41985)
0928 02:39:11.948688 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:11.948696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:11.948723 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:39:11.949246 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:11.949265 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:11.949312 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:11.949446 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:11.950220 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:11.951372 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:11.952501 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:11.953064 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:11.953191 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:11.955781 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:11.955784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:11.955785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:11.955796 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:11.957732 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:11.958108 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:11.958110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:11.958111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:11.958228 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:11.958427 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:11.958428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:11.958428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:11.958728 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:11.958891 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:11.958967 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:39:11.959103 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260928 02:39:11.971160 15348 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:11.972199 15348 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:11.984071 15348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa37144c0d75411c8d63d9ff28d61c45: Generated new TSK 0
I20260928 02:39:11.984274 15348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:39:12.016721 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41985
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:12.095249 15363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:12.095453 15363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:12.095477 15363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:12.096572 15363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:12.096604 15363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:12.096630 15363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:12.098223 15363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:12.098417 15363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:12.098590 15363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:39:12.103628 15363 server_base.cc:1034] running on GCE node
W20260928 02:39:12.103737 15368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:12.103636 15371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:12.103645 15369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:12.104118 15363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:12.104350 15363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:12.105507 15363 hybrid_clock.cc:648] HybridClock initialized: now 1790563152105506 us; error 41 us; skew 500 ppm
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:12.107959 15363 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:39:12.108358 15363 webserver.cc:460] Webserver started at http://127.9.130.1:39817/ using document root <none> and password file <none>
I20260928 02:39:12.108585 15363 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:12.108647 15363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:12.108774 15363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:12.109768 15363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "a85ac94bdb6d49cfb953c99813762208"
format_stamp: "Formatted at 2026-09-28 02:39:12 on dist-test-slave-kkrh"
server_key: "09291968ecf362cf7f4ceca40d4774f7"
server_key_iv: "259588d9f489cc87e8b365c672a9f35d"
server_key_version: "encryptionkey@0"
I20260928 02:39:12.110122 15363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "a85ac94bdb6d49cfb953c99813762208"
format_stamp: "Formatted at 2026-09-28 02:39:12 on dist-test-slave-kkrh"
server_key: "09291968ecf362cf7f4ceca40d4774f7"
server_key_iv: "259588d9f489cc87e8b365c672a9f35d"
server_key_version: "encryptionkey@0"
I20260928 02:39:12.111457 15363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:39:12.112298 15378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:12.112457 15363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:12.112550 15363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "a85ac94bdb6d49cfb953c99813762208"
format_stamp: "Formatted at 2026-09-28 02:39:12 on dist-test-slave-kkrh"
server_key: "09291968ecf362cf7f4ceca40d4774f7"
server_key_iv: "259588d9f489cc87e8b365c672a9f35d"
server_key_version: "encryptionkey@0"
I20260928 02:39:12.112624 15363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:12.118676 15363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:12.120529 15363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:12.120671 15363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:12.120862 15363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:12.121145 15363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:12.121193 15363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:12.121234 15363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:12.121263 15363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:12.127182 15363 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:46075
I20260928 02:39:12.127223 15491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:46075 every 8 connection(s)
I20260928 02:39:12.127511 15363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:39:12.131192 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 15363
I20260928 02:39:12.131305 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:39:12.131534 9736 external_mini_cluster.cc:1428] Setting key 23033342c6d948e55566c68e276d5edd
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:12.137010 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:12.128187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:33313 (local address 127.9.130.62:41985)
0928 02:39:12.128230 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:12.128232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:12.128397 (+ 165us) server_negotiation.cc:407] Connection header received
0928 02:39:12.129287 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:12.129288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:12.129303 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:12.129379 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:12.129916 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.130457 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.131513 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.131890 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.131970 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:12.134146 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:12.134147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:12.134148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:12.134157 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:12.135526 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:12.136042 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:12.136043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:12.136044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:12.136095 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:12.136371 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:12.136371 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:12.136371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:12.136483 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:12.136532 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:12.136867 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 02:39:12.136939 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:39:12.137307 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:12.128171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:41985 (local address 127.9.130.1:33313)
0928 02:39:12.128301 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:12.128333 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:12.129194 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:12.129432 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:12.129435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:12.129668 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:12.129843 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:12.129847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.130537 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.130537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:12.131425 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:12.131427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.132003 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.132004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:12.132112 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:12.132690 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:12.132704 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:12.134037 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:12.135619 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:12.135626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:12.135628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:12.135971 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:12.136161 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:12.136162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:12.136162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:12.136308 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:12.136547 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:12.136553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:12.136729 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 02:39:12.136855 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 02:39:12.137030 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
I20260928 02:39:12.137964 15492 heartbeater.cc:347] Connected to a master server at 127.9.130.62:41985
I20260928 02:39:12.138062 15492 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:12.138259 15492 heartbeater.cc:511] Master 127.9.130.62:41985 requested a full tablet report, sending...
I20260928 02:39:12.138867 15301 ts_manager.cc:194] Registered new tserver with Master: a85ac94bdb6d49cfb953c99813762208 (127.9.130.1:46075)
I20260928 02:39:12.139863 15301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:33313
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:39:12.158406 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41985
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:12.242012 15497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:12.242210 15497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:12.242233 15497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:12.243441 15497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:12.243484 15497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:12.243510 15497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:12.244906 15497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:12.245087 15497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:12.245249 15497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:12.250447 15502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:12.250517 15503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:12.250638 15497 server_base.cc:1034] running on GCE node
W20260928 02:39:12.250425 15505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:12.250916 15497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:12.251107 15497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:12.252240 15497 hybrid_clock.cc:648] HybridClock initialized: now 1790563152252210 us; error 50 us; skew 500 ppm
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:12.254856 15497 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:39:12.255306 15497 webserver.cc:460] Webserver started at http://127.9.130.2:38879/ using document root <none> and password file <none>
I20260928 02:39:12.255539 15497 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:12.255609 15497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:12.255757 15497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:12.256881 15497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "bd2804ff74624895bd2d9cbe0085250a"
format_stamp: "Formatted at 2026-09-28 02:39:12 on dist-test-slave-kkrh"
server_key: "d0bc15ed576910b86531039b78bd5733"
server_key_iv: "cb1f6195e6cb4049ae141830b273a668"
server_key_version: "encryptionkey@0"
I20260928 02:39:12.257244 15497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "bd2804ff74624895bd2d9cbe0085250a"
format_stamp: "Formatted at 2026-09-28 02:39:12 on dist-test-slave-kkrh"
server_key: "d0bc15ed576910b86531039b78bd5733"
server_key_iv: "cb1f6195e6cb4049ae141830b273a668"
server_key_version: "encryptionkey@0"
I20260928 02:39:12.258579 15497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:12.259394 15512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:12.259613 15497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:12.259718 15497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "bd2804ff74624895bd2d9cbe0085250a"
format_stamp: "Formatted at 2026-09-28 02:39:12 on dist-test-slave-kkrh"
server_key: "d0bc15ed576910b86531039b78bd5733"
server_key_iv: "cb1f6195e6cb4049ae141830b273a668"
server_key_version: "encryptionkey@0"
I20260928 02:39:12.259804 15497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:12.273007 15497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:12.274967 15497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:12.275132 15497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:12.275436 15497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:12.275758 15497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:12.275815 15497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:12.275902 15497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:12.275929 15497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:12.281582 15497 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:34833
I20260928 02:39:12.281631 15625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:34833 every 8 connection(s)
I20260928 02:39:12.281917 15497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:39:12.283041 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 15497
I20260928 02:39:12.283156 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:39:12.283346 9736 external_mini_cluster.cc:1428] Setting key fa963fc77d433a924f1b29b152977d19
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
I20260928 02:39:12.292075 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:12.282635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:34201 (local address 127.9.130.62:41985)
0928 02:39:12.282694 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:12.282696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:12.282892 (+ 196us) server_negotiation.cc:407] Connection header received
0928 02:39:12.283980 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:12.283981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:12.283991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:12.284068 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:12.284694 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.285195 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.286737 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.286946 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.287018 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:12.289259 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:12.289261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:12.289261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:12.289269 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:12.290635 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:12.291066 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:12.291068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:12.291068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:12.291127 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:12.291410 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:12.291411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:12.291412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:12.291528 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:12.291606 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:12.291901 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 02:39:12.291986 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:39:12.292258 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:12.282634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:41985 (local address 127.9.130.2:34201)
0928 02:39:12.282777 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:12.282808 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:12.283869 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:12.284153 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:12.284157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:12.284419 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:12.284600 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:12.284604 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.285291 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.285292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:12.286645 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:12.286647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.287035 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.287037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:12.287165 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:12.287737 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:12.287745 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:12.289165 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:12.290734 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:12.290740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:12.290742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:12.290974 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:12.291194 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:12.291195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:12.291195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:12.291334 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:12.291597 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:12.291603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:12.291739 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 02:39:12.291878 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:39:12.292037 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260928 02:39:12.292727 15626 heartbeater.cc:347] Connected to a master server at 127.9.130.62:41985
I20260928 02:39:12.292801 15626 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:12.292932 15626 heartbeater.cc:511] Master 127.9.130.62:41985 requested a full tablet report, sending...
I20260928 02:39:12.293589 15301 ts_manager.cc:194] Registered new tserver with Master: bd2804ff74624895bd2d9cbe0085250a (127.9.130.2:34833)
I20260928 02:39:12.293916 15301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:34201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:39:12.308807 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41985
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:12.395418 15631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:12.395642 15631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:12.395679 15631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:12.396873 15631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:12.396940 15631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:12.396984 15631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:12.398655 15631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:12.398928 15631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:12.399150 15631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:12.404058 15636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:12.404038 15637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:12.404040 15639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:12.404136 15631 server_base.cc:1034] running on GCE node
I20260928 02:39:12.404443 15631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:12.404629 15631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:12.405797 15631 hybrid_clock.cc:648] HybridClock initialized: now 1790563152405783 us; error 27 us; skew 500 ppm
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:12.408458 15631 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:39:12.408916 15631 webserver.cc:460] Webserver started at http://127.9.130.3:39377/ using document root <none> and password file <none>
I20260928 02:39:12.409194 15631 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:12.409266 15631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:12.409411 15631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:12.410389 15631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "432b795d9a4549019285cc4b5b03bcaf"
format_stamp: "Formatted at 2026-09-28 02:39:12 on dist-test-slave-kkrh"
server_key: "8dd71295184d30190e2961bc8789f4bd"
server_key_iv: "b850dda87053714df8cdc24b172650ac"
server_key_version: "encryptionkey@0"
I20260928 02:39:12.410738 15631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "432b795d9a4549019285cc4b5b03bcaf"
format_stamp: "Formatted at 2026-09-28 02:39:12 on dist-test-slave-kkrh"
server_key: "8dd71295184d30190e2961bc8789f4bd"
server_key_iv: "b850dda87053714df8cdc24b172650ac"
server_key_version: "encryptionkey@0"
I20260928 02:39:12.412629 15631 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 02:39:12.413571 15646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:12.413771 15631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:12.413883 15631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "432b795d9a4549019285cc4b5b03bcaf"
format_stamp: "Formatted at 2026-09-28 02:39:12 on dist-test-slave-kkrh"
server_key: "8dd71295184d30190e2961bc8789f4bd"
server_key_iv: "b850dda87053714df8cdc24b172650ac"
server_key_version: "encryptionkey@0"
I20260928 02:39:12.413980 15631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:12.435132 15631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:12.437587 15631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:12.437752 15631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:12.437994 15631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:12.438306 15631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:12.438362 15631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:12.438405 15631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:12.438433 15631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:12.444671 15631 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:35027
I20260928 02:39:12.444721 15759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:35027 every 8 connection(s)
I20260928 02:39:12.445060 15631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:12.453752 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 15631
I20260928 02:39:12.453846 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:39:12.454025 9736 external_mini_cluster.cc:1428] Setting key a7fd38bf32671a3324034b96ada3de97
I20260928 02:39:12.454469 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:12.445785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:42495 (local address 127.9.130.62:41985)
0928 02:39:12.445832 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:12.445834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:12.446020 (+ 186us) server_negotiation.cc:407] Connection header received
0928 02:39:12.446970 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:12.446971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:12.446982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:12.447085 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:12.447655 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.448256 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.449217 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.449401 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.449457 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:12.451443 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:12.451444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:12.451445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:12.451451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:12.453013 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:12.453354 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:12.453355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:12.453355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:12.453415 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:12.453679 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:12.453679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:12.453680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:12.453775 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:12.453893 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:12.454270 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 02:39:12.454366 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:39:12.454655 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:12.445751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:41985 (local address 127.9.130.3:42495)
0928 02:39:12.445905 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:12.445945 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:12.446863 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:12.447116 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:12.447119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:12.447360 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:12.447559 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:12.447563 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.448333 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.448334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:12.449137 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:12.449139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.449480 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.449481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:12.449578 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:12.449988 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:12.449996 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:12.451366 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:12.453078 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:12.453083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:12.453084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:12.453287 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:12.453474 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:12.453475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:12.453475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:12.453603 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:12.453862 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:12.453867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:12.454076 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 02:39:12.454229 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 02:39:12.454405 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260928 02:39:12.455020 15760 heartbeater.cc:347] Connected to a master server at 127.9.130.62:41985
I20260928 02:39:12.455112 15760 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:12.455241 15760 heartbeater.cc:511] Master 127.9.130.62:41985 requested a full tablet report, sending...
I20260928 02:39:12.455564 15300 ts_manager.cc:194] Registered new tserver with Master: 432b795d9a4549019285cc4b5b03bcaf (127.9.130.3:35027)
I20260928 02:39:12.456017 15300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:42495
I20260928 02:39:12.465876 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/test-admin.keytab.
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:12.479964 9736 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 02:39:12.481518 15767 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:12.480770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.9.130.1:46075)
0928 02:39:12.480937 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:12.480942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:12.480952 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:39:12.480983 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:12.480985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:12.480998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:12.481063 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:12.481432 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39692: BlockingRecv error: recv got EOF from 127.0.0.1:39692 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
W20260928 02:39:12.481590 9736 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:12.482585 9736 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:12.483570 9736 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.1@KRBTEST.COM
I20260928 02:39:12.493996 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:12.484191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.9.130.1:46075)
0928 02:39:12.484252 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:12.484254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:12.484264 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:39:12.484296 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:12.484297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:12.484306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:12.484355 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:12.484784 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.485478 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.486739 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.487279 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.487361 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:12.490335 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:12.490342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:12.490343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:12.490350 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:12.492321 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:12.492820 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:12.492824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:12.492824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:12.492896 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:12.493119 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:12.493120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:12.493120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:12.493328 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:12.493464 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:12.493720 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 02:39:12.493869 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 02:39:12.496344 15767 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:12.495431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39702 (local address 127.9.130.1:46075)
0928 02:39:12.495470 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:12.495486 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:12.495737 (+ 251us) server_negotiation.cc:407] Connection header received
0928 02:39:12.495869 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:12.495870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:12.495881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:12.495936 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:12.496291 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39702: BlockingRecv error: recv got EOF from 127.0.0.1:39702 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 02:39:12.496421 9736 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:12.497375 9736 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:12.498301 9736 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.1@KRBTEST.COM
I20260928 02:39:12.505275 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:12.498837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.9.130.1:46075)
0928 02:39:12.498896 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:12.498898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:12.498952 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:39:12.499026 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:12.499027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:12.499038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:12.499092 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:12.499496 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.500143 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.500936 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.501169 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.501261 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:12.503212 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:12.503213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:12.503213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:12.503219 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:12.504455 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:12.504745 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:12.504745 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:12.504745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:12.504791 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:12.504941 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:12.504941 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:12.504942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:12.505016 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:12.505079 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:12.505169 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 02:39:12.505220 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 02:39:12.506742 15767 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:12.506206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.9.130.1:46075)
0928 02:39:12.506259 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:12.506261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:12.506298 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:39:12.506369 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:12.506370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:12.506379 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:12.506447 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:12.506695 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39714: BlockingRecv error: recv got EOF from 127.0.0.1:39714 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 02:39:12.506907 9736 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563127283700-9736-0/krb5kdc/krb5cc)
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:12.507850 9736 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:12.508667 9736 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.1@KRBTEST.COM
I20260928 02:39:12.514740 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:12.509093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.9.130.1:46075)
0928 02:39:12.509128 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:12.509129 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:12.509190 (+ 61us) server_negotiation.cc:407] Connection header received
0928 02:39:12.509209 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:12.509210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:12.509216 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:12.509259 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:12.509579 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.509997 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.510646 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:12.510816 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:12.510869 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:12.512860 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:12.512861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:12.512862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:12.512867 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:12.513954 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:12.514231 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:12.514231 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:12.514232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:12.514277 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:12.514414 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:12.514415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:12.514415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:12.514486 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:12.514589 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:12.514637 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:39:12.514689 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:39:12.515432 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 15363
I20260928 02:39:12.520274 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 15497
I20260928 02:39:12.525002 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 15631
I20260928 02:39:12.529258 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 15269
2026-09-28T02:39:12Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1858 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15780](info): set up 2 sockets
Sep 28 02:39:12 dist-test-slave-kkrh krb5kdc[15780](info): commencing operation
krb5kdc: starting...
W20260928 02:39:13.568053 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:39:13 dist-test-slave-kkrh krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:39:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.62.keytab.
Entry for principal oryx/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.62.keytab.
I20260928 02:39:13.684000 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:39455
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.62.keytab
--principal=oryx/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:39455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:13.762604 15793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:13.762838 15793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:13.762876 15793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:13.764039 15793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:13.764103 15793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:13.764127 15793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:13.764156 15793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:13.764189 15793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:13.765635 15793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:39455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.62.keytab
--principal=oryx/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:39455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:13.765894 15793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:13.766109 15793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:13.770877 15799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:13.770896 15798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:13.770889 15801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:13.771178 15793 server_base.cc:1034] running on GCE node
I20260928 02:39:13.771440 15793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:13.771759 15793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:13.772924 15793 hybrid_clock.cc:648] HybridClock initialized: now 1790563153772900 us; error 38 us; skew 500 ppm
Sep 28 02:39:13 dist-test-slave-kkrh krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563153, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:13.775362 15793 init.cc:377] Logged in from keytab as oryx/127.9.130.62@KRBTEST.COM (short username oryx)
I20260928 02:39:13.775774 15793 webserver.cc:460] Webserver started at http://127.9.130.62:35635/ using document root <none> and password file <none>
I20260928 02:39:13.776028 15793 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:13.776098 15793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:13.776244 15793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:13.777243 15793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "973bc1b3988a4e6ab2347fad49023e0b"
format_stamp: "Formatted at 2026-09-28 02:39:13 on dist-test-slave-kkrh"
server_key: "0b9fa4c08653c1860271696b43765a67"
server_key_iv: "333f2eb0c25e7846c578796887943cd2"
server_key_version: "encryptionkey@0"
I20260928 02:39:13.777616 15793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "973bc1b3988a4e6ab2347fad49023e0b"
format_stamp: "Formatted at 2026-09-28 02:39:13 on dist-test-slave-kkrh"
server_key: "0b9fa4c08653c1860271696b43765a67"
server_key_iv: "333f2eb0c25e7846c578796887943cd2"
server_key_version: "encryptionkey@0"
I20260928 02:39:13.778851 15793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:39:13.779497 15808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:13.779664 15793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:13.779762 15793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "973bc1b3988a4e6ab2347fad49023e0b"
format_stamp: "Formatted at 2026-09-28 02:39:13 on dist-test-slave-kkrh"
server_key: "0b9fa4c08653c1860271696b43765a67"
server_key_iv: "333f2eb0c25e7846c578796887943cd2"
server_key_version: "encryptionkey@0"
I20260928 02:39:13.779862 15793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:13.793299 15793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:13.795531 15793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:13.795692 15793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:13.799376 15793 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:39455
I20260928 02:39:13.799420 15860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:39455 every 8 connection(s)
I20260928 02:39:13.799788 15793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:13.800730 15861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:13.802961 15861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b: Bootstrap starting.
I20260928 02:39:13.803508 15861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:13.803781 15861 log.cc:824] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:13.804497 15861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b: No bootstrap required, opened a new log
I20260928 02:39:13.806002 15861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } }
I20260928 02:39:13.806133 15861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:13.806154 15861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973bc1b3988a4e6ab2347fad49023e0b, State: Initialized, Role: FOLLOWER
I20260928 02:39:13.806366 15861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } }
I20260928 02:39:13.806452 15861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:13.806506 15861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:13.806556 15861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:13.807301 15861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } }
I20260928 02:39:13.807406 15861 leader_election.cc:302] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 973bc1b3988a4e6ab2347fad49023e0b; no voters:
I20260928 02:39:13.807566 15861 leader_election.cc:288] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:13.807595 15864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:13.807744 15864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [term 1 LEADER]: Becoming Leader. State: Replica: 973bc1b3988a4e6ab2347fad49023e0b, State: Running, Role: LEADER
I20260928 02:39:13.807832 15861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:13.807871 15864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } }
I20260928 02:39:13.808144 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 15793
I20260928 02:39:13.808270 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:13.808475 9736 external_mini_cluster.cc:1428] Setting key 21b58eeaac79ebac285b4341695c704d
I20260928 02:39:13.809118 15865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } }
I20260928 02:39:13.809128 15866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 973bc1b3988a4e6ab2347fad49023e0b. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } }
I20260928 02:39:13.809396 15865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:13.809402 15866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:13.809787 15880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:13.810217 15880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:13.811858 15880 catalog_manager.cc:1347] Generated new cluster ID: a65f718419f741969313ea95bf098fd5
I20260928 02:39:13.811909 15880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:39:13 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
I20260928 02:39:13.820454 15880 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:13.820945 15880 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:13.820990 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:13.809669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.9.130.62:39455)
0928 02:39:13.809869 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:13.809879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:13.809909 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:39:13.810007 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:13.810013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:13.810079 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:13.810228 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:13.811021 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:13.812328 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:13.813371 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:13.813957 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:13.814046 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:13.816756 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:13.816760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:13.816761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:13.816772 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:13.819093 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:13.819500 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:13.819504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:13.819504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:13.819623 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:13.819875 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:13.819876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:13.819876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:13.820143 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:13.820333 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:13.820459 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:39:13.820782 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
I20260928 02:39:13.827632 15880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b: Generated new TSK 0
I20260928 02:39:13.827791 15880 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab.
Entry for principal oryx/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab.
I20260928 02:39:13.858325 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:39455
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab
--principal=oryx/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:13.937345 15887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:13.937567 15887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:13.937603 15887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:13.938728 15887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:13.938795 15887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:13.938836 15887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:13.940297 15887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab
--principal=oryx/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:39455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:13.940550 15887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:13.940773 15887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:13.945726 15895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:13.945845 15887 server_base.cc:1034] running on GCE node
W20260928 02:39:13.945757 15893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:13.945957 15892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:13.946187 15887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:13.946403 15887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:13.947546 15887 hybrid_clock.cc:648] HybridClock initialized: now 1790563153947534 us; error 29 us; skew 500 ppm
Sep 28 02:39:13 dist-test-slave-kkrh krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563153, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:13.950078 15887 init.cc:377] Logged in from keytab as oryx/127.9.130.1@KRBTEST.COM (short username oryx)
I20260928 02:39:13.950500 15887 webserver.cc:460] Webserver started at http://127.9.130.1:33391/ using document root <none> and password file <none>
I20260928 02:39:13.950723 15887 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:13.950798 15887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:13.950944 15887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:13.952109 15887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "9c06bd0797244592ae7e5370e71d7166"
format_stamp: "Formatted at 2026-09-28 02:39:13 on dist-test-slave-kkrh"
server_key: "894769520af9af852602e77e60876050"
server_key_iv: "eab5557d93fd3f1e1d8bf0cf94ec4809"
server_key_version: "encryptionkey@0"
I20260928 02:39:13.952466 15887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "9c06bd0797244592ae7e5370e71d7166"
format_stamp: "Formatted at 2026-09-28 02:39:13 on dist-test-slave-kkrh"
server_key: "894769520af9af852602e77e60876050"
server_key_iv: "eab5557d93fd3f1e1d8bf0cf94ec4809"
server_key_version: "encryptionkey@0"
I20260928 02:39:13.953784 15887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:13.954476 15902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:13.954638 15887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:13.954759 15887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "9c06bd0797244592ae7e5370e71d7166"
format_stamp: "Formatted at 2026-09-28 02:39:13 on dist-test-slave-kkrh"
server_key: "894769520af9af852602e77e60876050"
server_key_iv: "eab5557d93fd3f1e1d8bf0cf94ec4809"
server_key_version: "encryptionkey@0"
I20260928 02:39:13.954834 15887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:13.975234 15887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:13.977231 15887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:13.977392 15887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:13.977618 15887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:13.977932 15887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:13.977986 15887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:13.978027 15887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:13.978055 15887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:13.984058 15887 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:43903
I20260928 02:39:13.984114 16015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:43903 every 8 connection(s)
I20260928 02:39:13.984470 15887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:39:13 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563153, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
I20260928 02:39:13.993345 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 15887
I20260928 02:39:13.993395 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:13.985225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:60829 (local address 127.9.130.62:39455)
0928 02:39:13.985304 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:13.985306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:13.985451 (+ 145us) server_negotiation.cc:407] Connection header received
0928 02:39:13.985907 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:13.985908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:13.985920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:13.985972 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:13.986861 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:13.987359 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:13.988296 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:13.988490 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:13.988540 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:13.990582 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:13.990583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:13.990583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:13.990589 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:13.991762 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:13.992292 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:13.992293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:13.992293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:13.992343 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:13.992634 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:13.992635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:13.992635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:13.992842 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:13.992912 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:13.993261 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 02:39:13.993332 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:39:13.993458 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:39:13.993666 9736 external_mini_cluster.cc:1428] Setting key a36d437820d385af0c28cd544aad4a7a
I20260928 02:39:13.993707 16018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:13.985176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.1:60829)
0928 02:39:13.985334 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:13.985369 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:13.985830 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:13.986024 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:13.986027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:13.986583 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:13.986776 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:13.986780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:13.987436 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:13.987437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:13.988229 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:13.988231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:13.988557 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:13.988558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:13.988659 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:13.989201 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:13.989209 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:13.990506 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:13.991996 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:13.992002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:13.992004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:13.992227 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:13.992429 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:13.992430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:13.992431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:13.992536 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:13.992930 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:13.992935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:13.993046 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 02:39:13.993234 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 02:39:13.993411 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260928 02:39:13.994195 16016 heartbeater.cc:347] Connected to a master server at 127.9.130.62:39455
I20260928 02:39:13.994279 16016 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:13.994480 16016 heartbeater.cc:511] Master 127.9.130.62:39455 requested a full tablet report, sending...
I20260928 02:39:13.995047 15823 ts_manager.cc:194] Registered new tserver with Master: 9c06bd0797244592ae7e5370e71d7166 (127.9.130.1:43903)
I20260928 02:39:13.995980 15823 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:60829
WARNING: no policy specified for oryx/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab.
Entry for principal oryx/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab.
I20260928 02:39:14.018213 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:39455
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab
--principal=oryx/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:14.098183 16021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:14.098407 16021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:14.098443 16021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:14.099574 16021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:14.099639 16021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:14.099681 16021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:14.101207 16021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab
--principal=oryx/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:39455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:14.101468 16021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:14.101732 16021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:14.106593 16027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:14.106611 16026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:14.106612 16029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:14.106950 16021 server_base.cc:1034] running on GCE node
I20260928 02:39:14.107103 16021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:14.107307 16021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:14.108444 16021 hybrid_clock.cc:648] HybridClock initialized: now 1790563154108432 us; error 28 us; skew 500 ppm
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:14.110863 16021 init.cc:377] Logged in from keytab as oryx/127.9.130.2@KRBTEST.COM (short username oryx)
I20260928 02:39:14.111302 16021 webserver.cc:460] Webserver started at http://127.9.130.2:42359/ using document root <none> and password file <none>
I20260928 02:39:14.111533 16021 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:14.111575 16021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:14.111717 16021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:14.112843 16021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "ab8c38454ade88f55ee9ea10527f6947"
server_key_iv: "1ef88bc8032fe75672899cc24b60ba4e"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.113200 16021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "ab8c38454ade88f55ee9ea10527f6947"
server_key_iv: "1ef88bc8032fe75672899cc24b60ba4e"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.114486 16021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:14.115448 16036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.115662 16021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:14.115725 16021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "ab8c38454ade88f55ee9ea10527f6947"
server_key_iv: "1ef88bc8032fe75672899cc24b60ba4e"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.115803 16021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:14.134101 16021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:14.135989 16021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:14.136135 16021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:14.136346 16021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:14.136641 16021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:14.136693 16021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.136735 16021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:14.136762 16021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.142413 16021 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:42421
I20260928 02:39:14.142480 16149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:42421 every 8 connection(s)
I20260928 02:39:14.142802 16021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
I20260928 02:39:14.151314 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.143529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:45715 (local address 127.9.130.62:39455)
0928 02:39:14.143585 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.143587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.143792 (+ 205us) server_negotiation.cc:407] Connection header received
0928 02:39:14.144315 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.144321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.144331 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.144383 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.145273 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.145771 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.146702 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.146880 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.146930 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.148971 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.148972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.148973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.148979 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.150230 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.150583 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.150584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.150584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.150632 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.150822 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.150822 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.150823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.150912 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.150957 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.151173 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.151253 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:39:14.151474 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.143514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.2:45715)
0928 02:39:14.143670 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.143703 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.144224 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.144434 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.144437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.144994 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.145185 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.145189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.145846 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.145847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.146624 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.146625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.146946 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.146947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.147038 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.147484 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.147492 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.148891 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.150303 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.150308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.150310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.150520 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.150689 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.150689 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.150690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.150780 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.150960 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.150964 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.151067 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.151144 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.151259 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260928 02:39:14.152009 16150 heartbeater.cc:347] Connected to a master server at 127.9.130.62:39455
I20260928 02:39:14.152084 16150 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:14.152272 16150 heartbeater.cc:511] Master 127.9.130.62:39455 requested a full tablet report, sending...
I20260928 02:39:14.152640 15823 ts_manager.cc:194] Registered new tserver with Master: 8d8dbeca3d3a4b0795204c4fa4a2badc (127.9.130.2:42421)
I20260928 02:39:14.152860 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 16021
I20260928 02:39:14.152974 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:39:14.153082 15823 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:45715
I20260928 02:39:14.153182 9736 external_mini_cluster.cc:1428] Setting key 81a6126f60f4a2df74c3c03a7855436d
WARNING: no policy specified for oryx/127.9.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab.
Entry for principal oryx/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab.
I20260928 02:39:14.178280 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:39455
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab
--principal=oryx/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:14.258925 16155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:14.259143 16155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:14.259179 16155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:14.260340 16155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:14.260406 16155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:14.260448 16155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:14.261924 16155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab
--principal=oryx/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:39455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:14.262189 16155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:14.262422 16155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:14.267318 16161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:14.267292 16163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:14.267520 16155 server_base.cc:1034] running on GCE node
W20260928 02:39:14.267567 16160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:14.267799 16155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:14.268076 16155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:14.269250 16155 hybrid_clock.cc:648] HybridClock initialized: now 1790563154269238 us; error 31 us; skew 500 ppm
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:14.271679 16155 init.cc:377] Logged in from keytab as oryx/127.9.130.3@KRBTEST.COM (short username oryx)
I20260928 02:39:14.272115 16155 webserver.cc:460] Webserver started at http://127.9.130.3:34209/ using document root <none> and password file <none>
I20260928 02:39:14.272342 16155 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:14.272385 16155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:14.272521 16155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:14.273535 16155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "e542c7fd344b4e1aa71cb8b91a0593eb"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "483de2c384b332dbdf12d3e27952ff95"
server_key_iv: "ffe1481ac6ed612151e86e3c4b46608a"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.273882 16155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "e542c7fd344b4e1aa71cb8b91a0593eb"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "483de2c384b332dbdf12d3e27952ff95"
server_key_iv: "ffe1481ac6ed612151e86e3c4b46608a"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.275137 16155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:39:14.275940 16170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.276116 16155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:14.276214 16155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "e542c7fd344b4e1aa71cb8b91a0593eb"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "483de2c384b332dbdf12d3e27952ff95"
server_key_iv: "ffe1481ac6ed612151e86e3c4b46608a"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.276290 16155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:14.309716 16155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:14.312212 16155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:14.312364 16155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:14.312613 16155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:14.312940 16155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:14.312994 16155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.313035 16155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:14.313061 16155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.318681 16155 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:35989
I20260928 02:39:14.318773 16283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:35989 every 8 connection(s)
I20260928 02:39:14.319080 16155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:39:14.322742 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 16155
I20260928 02:39:14.322849 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:39:14.323005 9736 external_mini_cluster.cc:1428] Setting key 6217c8e9ae9918f1f538f9c85378d5bf
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
I20260928 02:39:14.328428 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.319846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:49273 (local address 127.9.130.62:39455)
0928 02:39:14.319897 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.319899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.320112 (+ 213us) server_negotiation.cc:407] Connection header received
0928 02:39:14.320582 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.320583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.320593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.320661 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.321677 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.322157 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.323259 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.323549 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.323617 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.325761 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.325762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.325763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.325769 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.326996 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.327356 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.327357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.327357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.327406 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.327701 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.327703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.327703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.327875 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.327964 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.328261 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.328339 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:39:14.328603 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.319796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.3:49273)
0928 02:39:14.319992 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.320026 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.320500 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.320759 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.320765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.321411 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.321596 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.321601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.322259 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.322260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.323153 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.323156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.323740 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.323741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.323898 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.324342 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.324350 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.325681 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.327069 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.327074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.327076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.327291 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.327478 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.327479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.327480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.327629 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.327958 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.327964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.328120 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.328233 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.328383 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260928 02:39:14.329071 16284 heartbeater.cc:347] Connected to a master server at 127.9.130.62:39455
I20260928 02:39:14.329146 16284 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:14.329325 16284 heartbeater.cc:511] Master 127.9.130.62:39455 requested a full tablet report, sending...
I20260928 02:39:14.329689 15823 ts_manager.cc:194] Registered new tserver with Master: e542c7fd344b4e1aa71cb8b91a0593eb (127.9.130.3:35989)
I20260928 02:39:14.330006 15823 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:49273
I20260928 02:39:14.334685 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.130.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.130.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.130.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.61.keytab.
Entry for principal oryx/127.9.130.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.61.keytab.
I20260928 02:39:14.359411 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.61:40895
--webserver_interface=127.9.130.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.61.keytab
--principal=oryx/127.9.130.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:39455,127.9.130.61:40895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:14.439379 16289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:14.439607 16289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:14.439641 16289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:14.440776 16289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:14.440845 16289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:14.440869 16289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:14.440897 16289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:14.440940 16289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:14.442363 16289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:39455,127.9.130.61:40895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.61.keytab
--principal=oryx/127.9.130.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.61:40895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.130.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
W20260928 02:39:14.442489 16289 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.130.62:39455,127.9.130.61:40895'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 02:39:14.442641 16289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:14.442857 16289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:14.447554 16295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:14.447563 16297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:14.447563 16294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:14.447888 16289 server_base.cc:1034] running on GCE node
I20260928 02:39:14.448164 16289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:14.448441 16289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:14.449620 16289 hybrid_clock.cc:648] HybridClock initialized: now 1790563154449613 us; error 37 us; skew 500 ppm
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:14.452078 16289 init.cc:377] Logged in from keytab as oryx/127.9.130.61@KRBTEST.COM (short username oryx)
I20260928 02:39:14.452526 16289 webserver.cc:460] Webserver started at http://127.9.130.61:40329/ using document root <none> and password file <none>
I20260928 02:39:14.452760 16289 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:14.452827 16289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:14.452972 16289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:14.453985 16289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/data/instance:
uuid: "cb1abc6d08c64bc9984156fc48da77be"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "b5eb822746947c9ae11fab039872f92b"
server_key_iv: "404820f08814c16458caedaba2edd0dc"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.454341 16289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/wal/instance:
uuid: "cb1abc6d08c64bc9984156fc48da77be"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "b5eb822746947c9ae11fab039872f92b"
server_key_iv: "404820f08814c16458caedaba2edd0dc"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.455612 16289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:39:14.456453 16304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.456624 16289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.456720 16289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/wal
uuid: "cb1abc6d08c64bc9984156fc48da77be"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "b5eb822746947c9ae11fab039872f92b"
server_key_iv: "404820f08814c16458caedaba2edd0dc"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.456805 16289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:14.466020 16289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:14.467804 16289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:14.468007 16289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:14.471549 16289 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.61:40895
I20260928 02:39:14.471581 16356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.61:40895 every 8 connection(s)
I20260928 02:39:14.471967 16289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/data/info.pb
I20260928 02:39:14.472923 16357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:14.474041 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 16289
I20260928 02:39:14.474815 16357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.61@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.130.61@KRBTEST.COM
I20260928 02:39:14.483982 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.475886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.9.130.62:39455)
0928 02:39:14.475933 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.475935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.476029 (+ 94us) server_negotiation.cc:407] Connection header received
0928 02:39:14.476112 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.476113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.476120 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.476163 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.476683 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.477161 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.478064 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.478273 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.478355 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.480572 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.480574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.480575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.480582 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.482324 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.482825 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.482826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.482827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.482888 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.483210 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.483211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.483211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.483415 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.483479 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.483769 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.483878 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:39:14.484129 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.475753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47006)
0928 02:39:14.475927 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.475970 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.476049 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.476202 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.476204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.476493 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.476579 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.476583 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.477298 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.477299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.477961 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.477963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.478364 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.478366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.478434 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.478984 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.478991 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.480482 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.482409 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.482413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.482415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.482757 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.482999 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.483000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.483000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.483144 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.483481 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.483487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.483641 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.483731 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.483902 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":121,"threads_started":1}
I20260928 02:39:14.484153 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.474644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.9.130.61:40895)
0928 02:39:14.474810 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.474822 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.474851 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:39:14.475323 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.475330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.475378 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.475539 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.476319 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.477157 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.477875 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.478179 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.478295 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.481073 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.481078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.481078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.481084 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.482902 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.483324 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.483327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.483328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.483380 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.483589 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.483590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.483590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.483724 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.483917 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.483996 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.484069 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260928 02:39:14.484993 16357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.61@KRBTEST.COM for oryx/127.9.130.61@KRBTEST.COM
I20260928 02:39:14.492659 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.485204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:40895 (local address 127.0.0.1:58150)
0928 02:39:14.485242 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.485253 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.485287 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.485417 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.485417 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.485517 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.485543 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.485543 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.486510 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.486511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.486971 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.486973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.487561 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.487562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.487641 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.488232 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.488237 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.489676 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.491525 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.491526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.491527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.491788 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.492051 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.492051 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.492052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.492106 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.492361 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.492361 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.492396 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.492462 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.492553 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 02:39:14.492710 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.485236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.9.130.61:40895)
0928 02:39:14.485264 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.485266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.485275 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:39:14.485345 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.485346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.485353 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.485413 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.485598 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.486404 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.487063 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.487470 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.487552 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.489789 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.489790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.489790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.489797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.491452 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.491916 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.491917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.491918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.491978 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.492166 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.492166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.492167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.492283 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.492369 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.492489 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.492601 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":13}
I20260928 02:39:14.494313 16357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: Bootstrap starting.
I20260928 02:39:14.495287 16357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:14.495576 16357 log.cc:824] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:14.496433 16357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: No bootstrap required, opened a new log
I20260928 02:39:14.497721 16357 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } }
I20260928 02:39:14.497829 16357 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:14.497851 16357 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb1abc6d08c64bc9984156fc48da77be, State: Initialized, Role: FOLLOWER
I20260928 02:39:14.497931 16357 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } }
I20260928 02:39:14.498147 16357 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:14.498134 16363 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } } }
I20260928 02:39:14.498252 16363 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:39:14.499787 16374 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:39:14.499879 16374 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:39:14.501226 16289 master_runner.cc:428] Detected that this master cb1abc6d08c64bc9984156fc48da77be is joining an existing cluster
I20260928 02:39:14.501288 16289 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.130.61:40895
I20260928 02:39:14.509156 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.502256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.9.130.62:39455)
0928 02:39:14.502300 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.502302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.502557 (+ 255us) server_negotiation.cc:407] Connection header received
0928 02:39:14.502650 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.502651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.502661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.502716 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.503056 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.503629 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.504830 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.505033 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.505116 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.506166 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.506167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.506167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.506174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.507819 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.508378 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.508379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.508380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.508435 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.508658 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.508659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.508659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.508771 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.508845 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.508985 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.509063 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:39:14.509279 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.502427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58160 (local address 127.9.130.61:40895)
0928 02:39:14.502466 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.502469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.502478 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:39:14.502502 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.502503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.502512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.502578 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.503067 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.503788 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.504807 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.505108 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.505175 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.506337 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.506338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.506338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.506345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.508105 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.508500 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.508500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.508501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.508558 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.508747 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.508748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.508748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.508853 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.508891 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.509095 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.509181 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:39:14.509533 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.502256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:40895 (local address 127.0.0.1:58160)
0928 02:39:14.502388 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.502399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.502452 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.502636 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.502637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.502900 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.502982 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.502984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.503944 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.503945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.504726 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.504728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.505222 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.505223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.505280 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.505831 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.505835 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.506257 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.508170 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.508171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.508171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.508416 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.508634 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.508634 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.508634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.508684 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.508964 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.508965 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.509001 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.509355 (+ 354us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.509436 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260928 02:39:14.509829 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.502226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47016)
0928 02:39:14.502387 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.502399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.502593 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.502727 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.502728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.502926 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.502978 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.502980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.503736 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.503737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.504731 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.504732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.505127 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.505128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.505196 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.505649 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.505654 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.506065 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.507939 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.507941 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.507941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.508307 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.508507 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.508508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.508509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.508595 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.508836 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.508837 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.508877 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.508960 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.509081 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563153, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.62@KRBTEST.COM for oryx/127.9.130.61@KRBTEST.COM
I20260928 02:39:14.518746 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.511904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:40895 (local address 127.0.0.1:58168)
0928 02:39:14.512010 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.512053 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.512120 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.512263 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.512265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.512491 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.512531 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.512532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.513358 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.513359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.513924 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.513925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.514352 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.514353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.514420 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.514809 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.514813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.516073 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.517754 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.517761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.517763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.517951 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.518164 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.518165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.518165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.518232 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.518470 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.518474 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.518512 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.518575 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.518656 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":65,"threads_started":1}
I20260928 02:39:14.518772 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.511907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.9.130.61:40895)
0928 02:39:14.511940 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.511942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.512118 (+ 176us) server_negotiation.cc:407] Connection header received
0928 02:39:14.512172 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.512173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.512182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.512241 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.512594 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.513285 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.513994 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.514282 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.514348 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.516146 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.516147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.516147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.516154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.517651 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.518018 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.518018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.518019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.518074 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.518294 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.518295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.518295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.518409 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.518473 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.518621 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.518690 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:39:14.519357 15823 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.130.61:40895: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be"
member_type: NON_VOTER
last_known_addr {
host: "127.9.130.61"
port: 40895
}
attrs {
promote: true
}
}
dest_uuid: "973bc1b3988a4e6ab2347fad49023e0b"
cas_config_opid_index: -1
I20260928 02:39:14.519671 15823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } }
I20260928 02:39:14.520190 16383 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cb1abc6d08c64bc9984156fc48da77be (127.9.130.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } }
I20260928 02:39:14.520475 16331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:14.521276 16383 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.130.61:40895
I20260928 02:39:14.521317 16331 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 FOLLOWER]: Refusing update from remote peer 973bc1b3988a4e6ab2347fad49023e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 02:39:14.521359 16383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } }
I20260928 02:39:14.521417 16383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:14.521510 16289 master.cc:537] Master@127.9.130.61:40895 shutting down...
I20260928 02:39:14.522207 16384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:39:14.523002 15810 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 02:39:14.523411 15810 consensus_peers.cc:487] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b -> Peer cb1abc6d08c64bc9984156fc48da77be (127.9.130.61:40895): Couldn't send request to peer cb1abc6d08c64bc9984156fc48da77be. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:39:14.524255 16289 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:14.524346 16289 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:14.524397 16289 tablet_replica.cc:330] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: stopping tablet replica
I20260928 02:39:14.536685 16289 master.cc:552] Master@127.9.130.61:40895 shutdown complete.
I20260928 02:39:14.536761 16289 master_runner.cc:305] Clearing existing system tablet
I20260928 02:39:14.536895 16289 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:14.536952 16289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:14.537545 16289 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:14.538046 16388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.538159 16289 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:14.538213 16289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.538239 16289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/wal
uuid: "cb1abc6d08c64bc9984156fc48da77be"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "b5eb822746947c9ae11fab039872f92b"
server_key_iv: "404820f08814c16458caedaba2edd0dc"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.538815 16289 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:14.540264 16289 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:39:14.540350 16289 log.cc:1196] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 02:39:14.540550 16289 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: Deleting consensus metadata
I20260928 02:39:14.540669 16289 master_runner.cc:315] Copying system tablet from 127.9.130.62:39455
I20260928 02:39:14.541296 16289 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: tablet copy: Beginning tablet copy session from remote peer at address 127.9.130.62:39455
I20260928 02:39:14.547420 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.541654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.9.130.62:39455)
0928 02:39:14.541706 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.541708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.541779 (+ 71us) server_negotiation.cc:407] Connection header received
0928 02:39:14.541861 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.541862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.541873 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.541930 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.542290 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.542886 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.543623 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.543928 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.544011 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.544989 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.544990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.544991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.544998 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.546316 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.546671 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.546672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.546672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.546725 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.546926 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.546927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.546928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.547038 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.547111 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.547259 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.547331 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 02:39:14.547441 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.541589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47028)
0928 02:39:14.541706 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.541728 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.541797 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.541934 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.541935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.542112 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.542168 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.542170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.542978 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.542979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.543538 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.543540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.544012 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.544013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.544079 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.544578 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.544583 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.544880 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.546413 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.546414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.546414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.546584 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.546797 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.546798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.546798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.546853 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.547102 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.547103 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.547145 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.547232 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.547333 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":74,"threads_started":1}
I20260928 02:39:14.547701 15845 tablet_copy_service.cc:140] P 973bc1b3988a4e6ab2347fad49023e0b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cb1abc6d08c64bc9984156fc48da77be ({username='oryx', principal='oryx/127.9.130.61@KRBTEST.COM'} at 127.0.0.1:47028)
I20260928 02:39:14.547804 15845 tablet_copy_service.cc:161] P 973bc1b3988a4e6ab2347fad49023e0b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cb1abc6d08c64bc9984156fc48da77be at {username='oryx', principal='oryx/127.9.130.61@KRBTEST.COM'} at 127.0.0.1:47028: session id = cb1abc6d08c64bc9984156fc48da77be-00000000000000000000000000000000
I20260928 02:39:14.548403 15845 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:39:14.549001 16289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:39:14.549899 16289 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: tablet copy: Starting download of 0 data blocks...
I20260928 02:39:14.550046 16289 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: tablet copy: Starting download of 1 WAL segments...
I20260928 02:39:14.550855 16289 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:39:14.551651 15845 tablet_copy_service.cc:342] P 973bc1b3988a4e6ab2347fad49023e0b: Request end of tablet copy session cb1abc6d08c64bc9984156fc48da77be-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.130.61@KRBTEST.COM'} at 127.0.0.1:47028
I20260928 02:39:14.551707 15845 tablet_copy_service.cc:433] P 973bc1b3988a4e6ab2347fad49023e0b: ending tablet copy session cb1abc6d08c64bc9984156fc48da77be-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cb1abc6d08c64bc9984156fc48da77be
I20260928 02:39:14.554175 16289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:14.555395 16396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:14.555395 16395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:14.555493 16398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:14.555573 16289 server_base.cc:1034] running on GCE node
I20260928 02:39:14.555732 16289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:14.555883 16289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:14.557008 16289 hybrid_clock.cc:648] HybridClock initialized: now 1790563154556999 us; error 22 us; skew 500 ppm
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:14.558265 16289 init.cc:377] Logged in from keytab as oryx/127.9.130.61@KRBTEST.COM (short username oryx)
I20260928 02:39:14.558511 16289 webserver.cc:460] Webserver started at http://127.9.130.61:35649/ using document root <none> and password file <none>
I20260928 02:39:14.558656 16289 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:14.558683 16289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:14.559247 16289 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:14.559708 16405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.559872 16289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.559916 16289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/wal
uuid: "cb1abc6d08c64bc9984156fc48da77be"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "b5eb822746947c9ae11fab039872f92b"
server_key_iv: "404820f08814c16458caedaba2edd0dc"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.559969 16289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:14.581244 16289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:14.582995 16289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:14.586138 16289 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.61:40895
I20260928 02:39:14.586164 16457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.61:40895 every 8 connection(s)
I20260928 02:39:14.586452 16289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-1/data/info.pb
I20260928 02:39:14.587285 16458 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:39:14.587862 16458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: Bootstrap starting.
I20260928 02:39:14.590754 16458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:39:14.590989 16458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: Bootstrap complete.
I20260928 02:39:14.591176 16458 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } }
I20260928 02:39:14.591259 16458 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cb1abc6d08c64bc9984156fc48da77be, State: Initialized, Role: LEARNER
I20260928 02:39:14.591301 16458 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } }
I20260928 02:39:14.591498 16458 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:14.591502 16461 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } }
I20260928 02:39:14.591657 16461 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: This master's current role is: LEARNER
I20260928 02:39:14.593024 16472 catalog_manager.cc:1259] Loaded cluster ID: a65f718419f741969313ea95bf098fd5
I20260928 02:39:14.593070 16472 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: loading cluster ID for follower catalog manager: success
I20260928 02:39:14.594182 16472 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: acquiring CA information for follower catalog manager: success
I20260928 02:39:14.594463 16472 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:39:14.642534 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.636576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58196 (local address 127.9.130.61:40895)
0928 02:39:14.636712 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.636714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.636722 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:14.636753 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.636754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.636762 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.636823 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.637378 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.638022 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.638710 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.638902 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.638958 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.640153 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.640154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.640154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.640161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.641605 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.641897 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.641897 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.641898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.641946 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.642107 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.642108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.642108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.642197 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.642268 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.642384 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.642440 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.9.130.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.130.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.130.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.60.keytab.
Entry for principal oryx/127.9.130.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.60.keytab.
I20260928 02:39:14.666851 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.60:34557
--webserver_interface=127.9.130.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.60.keytab
--principal=oryx/127.9.130.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:39455,127.9.130.61:40895,127.9.130.60:34557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:14.754552 16476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:14.754791 16476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:14.754817 16476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:14.756060 16476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:14.756104 16476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:14.756177 16476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:14.756191 16476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:14.756206 16476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:14.757746 16476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:39455,127.9.130.61:40895,127.9.130.60:34557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.60.keytab
--principal=oryx/127.9.130.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.60:34557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.130.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:14.757994 16476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:14.758227 16476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:14.763262 16482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:14.763260 16481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:14.763348 16484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:14.763408 16476 server_base.cc:1034] running on GCE node
I20260928 02:39:14.763762 16476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:14.764046 16476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:14.765184 16476 hybrid_clock.cc:648] HybridClock initialized: now 1790563154765165 us; error 37 us; skew 500 ppm
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:14.768041 16476 init.cc:377] Logged in from keytab as oryx/127.9.130.60@KRBTEST.COM (short username oryx)
I20260928 02:39:14.768447 16476 webserver.cc:460] Webserver started at http://127.9.130.60:37617/ using document root <none> and password file <none>
I20260928 02:39:14.768677 16476 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:14.768728 16476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:14.768877 16476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:14.769877 16476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/data/instance:
uuid: "9c08a8244eaa49f5974db0eb4be56852"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "cac530961e0f1b759eb74176e68e7afc"
server_key_iv: "0eb9b7e949e57d2c2b55acf4d1bb37c2"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.770219 16476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/wal/instance:
uuid: "9c08a8244eaa49f5974db0eb4be56852"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "cac530961e0f1b759eb74176e68e7afc"
server_key_iv: "0eb9b7e949e57d2c2b55acf4d1bb37c2"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.771505 16476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 02:39:14.772284 16491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.772475 16476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.772558 16476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/wal
uuid: "9c08a8244eaa49f5974db0eb4be56852"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "cac530961e0f1b759eb74176e68e7afc"
server_key_iv: "0eb9b7e949e57d2c2b55acf4d1bb37c2"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.772627 16476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:14.790097 16476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:14.792309 16476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:14.792466 16476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:14.796284 16476 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.60:34557
I20260928 02:39:14.796341 16543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.60:34557 every 8 connection(s)
I20260928 02:39:14.796676 16476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/data/info.pb
I20260928 02:39:14.797647 16544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:14.799129 16544 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:39:14.801441 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 16476
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.60@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.130.60@KRBTEST.COM
I20260928 02:39:14.810261 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.799907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.9.130.62:39455)
0928 02:39:14.799969 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.799972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.800156 (+ 184us) server_negotiation.cc:407] Connection header received
0928 02:39:14.801313 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.801314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.801327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.801400 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.802161 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.802710 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.803855 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.804125 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.804215 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.806547 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.806548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.806549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.806555 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.808422 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.809001 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.809002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.809002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.809071 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.809447 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.809448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.809449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.809605 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.809695 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.810058 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.810144 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:39:14.810528 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.799790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47030)
0928 02:39:14.800025 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.800066 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.801096 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.801497 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.801502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.801824 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.802080 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.802085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.802820 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.802821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.803719 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.803721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.804213 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.804214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.804347 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.804853 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.804861 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.806451 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.808505 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.808511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.808513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.808898 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.809244 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.809245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.809245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.809371 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.809717 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.809725 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.809880 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.810018 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.810223 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":16,"thread_start_us":132,"threads_started":1}
I20260928 02:39:14.811120 16544 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:39:14.811372 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.801922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.9.130.60:34557)
0928 02:39:14.802126 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.802130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.802147 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:39:14.802172 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.802174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.802194 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.802268 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.802864 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.803746 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.804533 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.804828 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.804903 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.807513 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.807520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.807521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.807530 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.809613 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.810090 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.810093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.810093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.810152 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.810430 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.810431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.810432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.810594 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.811115 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.811156 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.811238 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":178,"thread_start_us":100,"threads_started":1}
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.60@KRBTEST.COM for oryx/127.9.130.61@KRBTEST.COM
I20260928 02:39:14.817330 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.811459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:40895 (local address 127.0.0.1:58202)
0928 02:39:14.811632 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.811643 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.811700 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.811875 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.811875 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.812029 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.812084 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.812086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.812698 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.812699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.813163 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.813164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.813545 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.813546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.813597 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.813916 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.813920 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.814985 (+ 1065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.816604 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.816604 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.816605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.816751 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.816916 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.816916 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.816917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.816962 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.817126 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.817127 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.817155 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.817203 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.817261 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260928 02:39:14.817387 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.811482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.9.130.61:40895)
0928 02:39:14.811531 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.811533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.811708 (+ 175us) server_negotiation.cc:407] Connection header received
0928 02:39:14.811766 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.811767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.811782 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.811874 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.812155 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.812616 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.813278 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.813477 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.813536 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.815066 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.815068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.815068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.815074 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.816535 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.816815 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.816816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.816816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.816861 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.816999 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.816999 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.816999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.817080 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.817122 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.817218 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.817328 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:39:14.817759 16544 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.60@KRBTEST.COM for oryx/127.9.130.60@KRBTEST.COM
I20260928 02:39:14.823376 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.818064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.9.130.60:34557)
0928 02:39:14.818115 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.818116 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.818124 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:14.818150 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.818150 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.818157 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.818207 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.818466 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.818918 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.819518 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.819715 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.819764 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.821203 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.821204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.821205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.821210 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.822566 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.822861 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.822861 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.822861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.822906 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.823044 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.823044 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.823045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.823123 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.823172 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.823253 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.823310 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:39:14.823482 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.818019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:34557 (local address 127.0.0.1:41794)
0928 02:39:14.818063 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.818073 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.818108 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.818212 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.818213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.818312 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.818336 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.818337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.819021 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.819022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.819452 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.819453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.819762 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.819763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.819801 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.820149 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.820153 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.821136 (+ 983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.822637 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.822638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.822638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.822784 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.822961 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.822962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.822962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.823008 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.823167 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.823167 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.823195 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.823239 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.823355 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":43}
I20260928 02:39:14.824728 16544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: Bootstrap starting.
I20260928 02:39:14.825544 16544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:14.825821 16544 log.cc:824] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:14.826483 16544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: No bootstrap required, opened a new log
I20260928 02:39:14.827890 16544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } }
I20260928 02:39:14.828037 16544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:14.828081 16544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c08a8244eaa49f5974db0eb4be56852, State: Initialized, Role: FOLLOWER
I20260928 02:39:14.828231 16544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } }
I20260928 02:39:14.828536 16544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:14.828521 16549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } } }
I20260928 02:39:14.828665 16549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:39:14.830356 16560 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:39:14.830420 16560 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:39:14.832679 16476 master_runner.cc:428] Detected that this master 9c08a8244eaa49f5974db0eb4be56852 is joining an existing cluster
I20260928 02:39:14.832726 16476 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.130.60:34557
I20260928 02:39:14.840685 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.833610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47034 (local address 127.9.130.62:39455)
0928 02:39:14.833721 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.833723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.833766 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:39:14.833868 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.833869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.833881 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.833934 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.834316 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.834863 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.836159 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.836493 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.836537 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.837662 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.837663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.837663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.837671 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.839314 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.839800 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.839801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.839801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.839892 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.840114 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.840115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.840115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.840253 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.840297 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.840493 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.840577 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 02:39:14.840791 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.833738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.9.130.60:34557)
0928 02:39:14.833768 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.833770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.833827 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:39:14.833888 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.833889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.833896 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.833943 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.834793 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.835283 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.836147 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.836336 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.836416 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.837514 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.837515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.837516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.837522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.839614 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.840038 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.840038 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.840039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.840081 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.840265 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.840265 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.840266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.840383 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.840424 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.840585 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.840683 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 02:39:14.840903 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.833574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:34557 (local address 127.0.0.1:41796)
0928 02:39:14.833770 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.833780 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.833841 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.833948 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.833949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.834207 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.834708 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.834711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.835372 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.835373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.836057 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.836059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.836431 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.836432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.836487 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.836970 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.836975 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.837414 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.839683 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.839683 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.839685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.839964 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.840146 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.840146 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.840147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.840201 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.840465 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.840465 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.840493 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.840658 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.840733 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260928 02:39:14.841361 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.833566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47034)
0928 02:39:14.833700 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.833710 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.833792 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.833976 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.833977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.834178 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.834236 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.834237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.834942 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.834943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.836059 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.836062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.836614 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.836615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.836677 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.837138 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.837143 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.837573 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.839422 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.839423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.839423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.839712 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.839975 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.839976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.839976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.840038 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.840342 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.840343 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.840377 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.840796 (+ 419us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.841272 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":148,"thread_start_us":83,"threads_started":1}
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563153, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.62@KRBTEST.COM for oryx/127.9.130.60@KRBTEST.COM
I20260928 02:39:14.862751 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 15887
I20260928 02:39:14.863914 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.843220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:34557 (local address 127.0.0.1:41800)
0928 02:39:14.843290 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.843301 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.843345 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.843475 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.843475 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.843619 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.843652 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.843653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.844355 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.844356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.844840 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.844842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.845195 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.845196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.845258 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.845655 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.845660 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.846789 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.862295 (+ 15506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.862299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.862300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.862539 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.862758 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.862759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.862759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.862817 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.863573 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.863574 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.863625 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.863672 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.863787 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 02:39:14.864611 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.843252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.9.130.60:34557)
0928 02:39:14.843287 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.843288 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.843331 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:39:14.843409 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.843410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.843418 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.843471 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.843732 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.844267 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.844939 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.845126 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.845196 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.846898 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.846899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.846900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.846906 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.862157 (+ 15251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.862629 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.862630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.862630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.862685 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.863332 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.863333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.863333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.863473 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.864407 (+ 934us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.864449 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.864525 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:39:14.865059 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.840035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.9.130.61:40895)
0928 02:39:14.840076 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.840078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.840305 (+ 227us) server_negotiation.cc:407] Connection header received
0928 02:39:14.840398 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.840399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.840411 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.840441 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.841598 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.842362 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.843489 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.843790 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.843854 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.861022 (+ 17168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.861024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.861025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.861034 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.862646 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.864164 (+ 1518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.864166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.864167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.864231 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.864450 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.864451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.864451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.864593 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.864634 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.864843 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.864936 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:39:14.865415 15823 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.130.60:34557: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852"
member_type: NON_VOTER
last_known_addr {
host: "127.9.130.60"
port: 34557
}
attrs {
promote: true
}
}
dest_uuid: "973bc1b3988a4e6ab2347fad49023e0b"
cas_config_opid_index: 5
I20260928 02:39:14.865679 15823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } }
I20260928 02:39:14.866055 16384 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9c08a8244eaa49f5974db0eb4be56852 (127.9.130.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } }
I20260928 02:39:14.866961 16384 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.130.60:34557
I20260928 02:39:14.867041 16384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } }
I20260928 02:39:14.867125 16384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:14.867668 16518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:14.868557 16518 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 FOLLOWER]: Refusing update from remote peer 973bc1b3988a4e6ab2347fad49023e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:39:14.868798 16383 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:14.869148 16476 master.cc:537] Master@127.9.130.60:34557 shutting down...
I20260928 02:39:14.869300 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:43903
--local_ip_for_outbound_sockets=127.9.130.1
--tserver_master_addrs=127.9.130.62:39455
--webserver_port=33391
--webserver_interface=127.9.130.1
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab
--principal=oryx/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.130.62:39455,127.9.130.61:40895
--tserver_master_addrs=127.9.130.62:39455,127.9.130.61:40895,127.9.130.60:34557 with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:39:14.871074 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.867218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.9.130.61:40895)
0928 02:39:14.867255 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.867257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.867266 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:39:14.867288 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.867289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.867321 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.867383 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:14.867587 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.868327 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.870184 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.870741 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.870818 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.870895 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.871002 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.871029 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:39:14.871114 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.866155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:40895 (local address 127.0.0.1:58222)
0928 02:39:14.866319 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.866331 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.866374 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.867443 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.867444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.867446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:14.867487 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.867489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.868477 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.868478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.870089 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.870092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.870818 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.870819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.870897 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.870923 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.870986 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.871030 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260928 02:39:14.871384 16432 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 LEARNER]: Refusing update from remote peer 973bc1b3988a4e6ab2347fad49023e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:39:14.871616 16518 raft_consensus.cc:858] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } }
W20260928 02:39:14.871682 16518 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } }
I20260928 02:39:14.871934 16383 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 02:39:14.872437 16432 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9c08a8244eaa49f5974db0eb4be56852 (127.9.130.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } }
I20260928 02:39:14.872505 16518 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cb1abc6d08c64bc9984156fc48da77be (127.9.130.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } }
I20260928 02:39:14.873489 16461 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 973bc1b3988a4e6ab2347fad49023e0b. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } }
I20260928 02:39:14.873560 16461 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: This master's current role is: LEARNER
I20260928 02:39:14.873589 16461 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } }
I20260928 02:39:14.873634 16461 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: This master's current role is: LEARNER
I20260928 02:39:14.874557 16549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 973bc1b3988a4e6ab2347fad49023e0b. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } }
I20260928 02:39:14.874624 16549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 02:39:14.875423 16383 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b: attempting to promote NON_VOTER cb1abc6d08c64bc9984156fc48da77be to VOTER
I20260928 02:39:14.875619 16383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } }
I20260928 02:39:14.876420 16432 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 LEARNER]: Refusing update from remote peer 973bc1b3988a4e6ab2347fad49023e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 02:39:14.876643 16384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 02:39:14.876713 16570 mvcc.cc:205] Tried to move back new op lower bound from 7334146678076633088 to 7334146678000017408. Current Snapshot: MvccSnapshot[applied={T|T < 7334146678010138624}]
I20260928 02:39:14.876894 16549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } }
I20260928 02:39:14.876953 16549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: This master's current role is: NON_PARTICIPANT
W20260928 02:39:14.877287 15811 consensus_peers.cc:487] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b -> Peer 9c08a8244eaa49f5974db0eb4be56852 (127.9.130.60:34557): Couldn't send request to peer 9c08a8244eaa49f5974db0eb4be56852. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:39:14.877585 16573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: NON_VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: true } } }
I20260928 02:39:14.877683 16573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 02:39:14.878103 16476 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260928 02:39:14.878202 16476 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260928 02:39:14.878242 16476 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: stopping tablet replica
I20260928 02:39:14.878487 16383 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cb1abc6d08c64bc9984156fc48da77be (127.9.130.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } }
I20260928 02:39:14.878662 16432 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cb1abc6d08c64bc9984156fc48da77be (127.9.130.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } }
I20260928 02:39:14.879370 16383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } }
I20260928 02:39:14.879451 16383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:14.879510 16383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } }
I20260928 02:39:14.879558 16383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:14.879561 16575 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } }
I20260928 02:39:14.879643 16575 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:39:14.880399 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.840119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:40895 (local address 127.0.0.1:58216)
0928 02:39:14.840235 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.840244 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.840328 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.840502 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.840503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.841329 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.841386 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.841388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.842460 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.842461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.843389 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.843391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.843927 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.843928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.843991 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.860426 (+ 16435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.860432 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.860886 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.862734 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.862735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.862735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.862980 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.864307 (+ 1327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.864307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.864308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.864379 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.864677 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.864678 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.864723 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.865179 (+ 456us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.865287 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260928 02:39:14.892896 16476 master.cc:552] Master@127.9.130.60:34557 shutdown complete.
I20260928 02:39:14.892958 16476 master_runner.cc:305] Clearing existing system tablet
I20260928 02:39:14.893131 16476 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:14.893181 16476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:14.893867 16476 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.894347 16581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.894469 16476 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:14.894516 16476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.894544 16476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/wal
uuid: "9c08a8244eaa49f5974db0eb4be56852"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "cac530961e0f1b759eb74176e68e7afc"
server_key_iv: "0eb9b7e949e57d2c2b55acf4d1bb37c2"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.895138 16476 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:14.896445 16476 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:39:14.896521 16476 log.cc:1196] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 02:39:14.896770 16476 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: Deleting consensus metadata
I20260928 02:39:14.896903 16476 master_runner.cc:315] Copying system tablet from 127.9.130.62:39455
I20260928 02:39:14.897516 16476 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: tablet copy: Beginning tablet copy session from remote peer at address 127.9.130.62:39455
I20260928 02:39:14.904224 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.897825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47040)
0928 02:39:14.897942 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:14.897952 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:14.898012 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:14.898145 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:14.898146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:14.898318 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:14.898354 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.898355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.899012 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.899013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.899590 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:14.899592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.900067 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.900068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:14.900128 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.900971 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:14.900976 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:14.901271 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:14.902934 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.902935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.902936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.903117 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.903410 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:14.903411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:14.903412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:14.903485 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:14.903743 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:14.903744 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:14.903789 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:39:14.903983 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 02:39:14.904093 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260928 02:39:14.904340 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:14.897879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47040 (local address 127.9.130.62:39455)
0928 02:39:14.897927 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:14.897930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:14.897997 (+ 67us) server_negotiation.cc:407] Connection header received
0928 02:39:14.898074 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:14.898075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:14.898084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:14.898141 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:14.898433 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.898932 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.899700 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:14.899984 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:14.900067 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:14.901384 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:14.901386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:14.901386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:14.901393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:14.902831 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.903273 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.903275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.903275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.903337 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:14.903550 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:14.903551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:14.903551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:14.903676 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:14.903753 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:14.904015 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 02:39:14.904226 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:39:14.904731 15845 tablet_copy_service.cc:140] P 973bc1b3988a4e6ab2347fad49023e0b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9c08a8244eaa49f5974db0eb4be56852 ({username='oryx', principal='oryx/127.9.130.60@KRBTEST.COM'} at 127.0.0.1:47040)
I20260928 02:39:14.904812 15845 tablet_copy_service.cc:161] P 973bc1b3988a4e6ab2347fad49023e0b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9c08a8244eaa49f5974db0eb4be56852 at {username='oryx', principal='oryx/127.9.130.60@KRBTEST.COM'} at 127.0.0.1:47040: session id = 9c08a8244eaa49f5974db0eb4be56852-00000000000000000000000000000000
I20260928 02:39:14.905252 15845 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:39:14.905711 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:39:14.906598 16476 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: tablet copy: Starting download of 0 data blocks...
I20260928 02:39:14.906785 16476 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: tablet copy: Starting download of 1 WAL segments...
I20260928 02:39:14.907434 16476 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:39:14.908327 15845 tablet_copy_service.cc:342] P 973bc1b3988a4e6ab2347fad49023e0b: Request end of tablet copy session 9c08a8244eaa49f5974db0eb4be56852-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.130.60@KRBTEST.COM'} at 127.0.0.1:47040
I20260928 02:39:14.908378 15845 tablet_copy_service.cc:433] P 973bc1b3988a4e6ab2347fad49023e0b: ending tablet copy session 9c08a8244eaa49f5974db0eb4be56852-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9c08a8244eaa49f5974db0eb4be56852
I20260928 02:39:14.910722 16476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:14.912041 16588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:14.912101 16476 server_base.cc:1034] running on GCE node
W20260928 02:39:14.912165 16591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:14.912389 16589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:14.912535 16476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:14.912670 16476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:14.913793 16476 hybrid_clock.cc:648] HybridClock initialized: now 1790563154913787 us; error 24 us; skew 500 ppm
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:14.915092 16476 init.cc:377] Logged in from keytab as oryx/127.9.130.60@KRBTEST.COM (short username oryx)
I20260928 02:39:14.915462 16476 webserver.cc:460] Webserver started at http://127.9.130.60:32929/ using document root <none> and password file <none>
I20260928 02:39:14.915577 16476 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:14.915612 16476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:14.916294 16476 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:14.916776 16598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.916919 16476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:14.916962 16476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/wal
uuid: "9c08a8244eaa49f5974db0eb4be56852"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "cac530961e0f1b759eb74176e68e7afc"
server_key_iv: "0eb9b7e949e57d2c2b55acf4d1bb37c2"
server_key_version: "encryptionkey@0"
I20260928 02:39:14.917021 16476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:14.922390 16476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:14.924116 16476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:14.927078 16476 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.60:34557
I20260928 02:39:14.927435 16476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/master-2/data/info.pb
I20260928 02:39:14.927698 16650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.60:34557 every 8 connection(s)
I20260928 02:39:14.928258 16651 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:39:14.928637 16651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: Bootstrap starting.
I20260928 02:39:14.930366 16651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:39:14.930574 16651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: Bootstrap complete.
I20260928 02:39:14.930778 16651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } }
I20260928 02:39:14.930861 16651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9c08a8244eaa49f5974db0eb4be56852, State: Initialized, Role: LEARNER
I20260928 02:39:14.930905 16651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } }
I20260928 02:39:14.931063 16651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:14.932456 16665 catalog_manager.cc:1259] Loaded cluster ID: a65f718419f741969313ea95bf098fd5
I20260928 02:39:14.932900 16665 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: loading cluster ID for follower catalog manager: success
I20260928 02:39:14.933746 16665 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: acquiring CA information for follower catalog manager: success
I20260928 02:39:14.934021 16665 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:39:14.935900 16654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } }
I20260928 02:39:14.936034 16654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: This master's current role is: LEARNER
W20260928 02:39:14.987371 16569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:14.987653 16569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:14.987690 16569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:14.989423 16569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:14.989485 16569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:14.989526 16569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:14.991691 16569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab
--principal=oryx/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:43903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=33391
--tserver_master_addrs=127.9.130.62:39455,127.9.130.61:40895,127.9.130.60:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:14.991999 16569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:14.992259 16569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:14.998445 16670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:14.998600 16569 server_base.cc:1034] running on GCE node
W20260928 02:39:14.998451 16671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:14.998459 16673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:14.998945 16569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:14.999191 16569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:15.000344 16569 hybrid_clock.cc:648] HybridClock initialized: now 1790563155000326 us; error 32 us; skew 500 ppm
Sep 28 02:39:14 dist-test-slave-kkrh krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:15.003136 16569 init.cc:377] Logged in from keytab as oryx/127.9.130.1@KRBTEST.COM (short username oryx)
I20260928 02:39:15.003636 16569 webserver.cc:460] Webserver started at http://127.9.130.1:33391/ using document root <none> and password file <none>
I20260928 02:39:15.003928 16569 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:15.004009 16569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:15.005493 16569 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:15.006464 16680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:15.006656 16569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:15.006771 16569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "9c06bd0797244592ae7e5370e71d7166"
format_stamp: "Formatted at 2026-09-28 02:39:13 on dist-test-slave-kkrh"
server_key: "894769520af9af852602e77e60876050"
server_key_iv: "eab5557d93fd3f1e1d8bf0cf94ec4809"
server_key_version: "encryptionkey@0"
I20260928 02:39:15.007117 16569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:15.016522 16569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:15.018885 16569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:15.019066 16569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:15.019296 16569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:15.019591 16569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:15.019642 16569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:15.019691 16569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:15.019718 16569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:15.026139 16569 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:43903
I20260928 02:39:15.026209 16793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:43903 every 8 connection(s)
I20260928 02:39:15.026552 16569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:39:15 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.60@KRBTEST.COM
Sep 28 02:39:15 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
I20260928 02:39:15.035239 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 16569
Sep 28 02:39:15 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563154, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.61@KRBTEST.COM
I20260928 02:39:15.035636 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 16021
I20260928 02:39:15.038955 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.027282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:34557 (local address 127.9.130.1:41201)
0928 02:39:15.027701 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.027712 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.028599 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.028889 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.028890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.029209 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:15.029439 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.029444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.030352 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.030353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.031490 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.031493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.032051 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.032052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.032504 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.033061 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:15.033066 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:15.034509 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:15.036973 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.036979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.036981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.037560 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.037781 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.037782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.037783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.037921 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.038213 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:15.038218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:15.038377 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.038479 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.038658 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":112,"thread_start_us":93,"threads_started":1}
I20260928 02:39:15.039453 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.027561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:41201 (local address 127.9.130.60:34557)
0928 02:39:15.027663 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.027665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.027782 (+ 117us) server_negotiation.cc:407] Connection header received
0928 02:39:15.028755 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.028756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.028769 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.028832 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.029542 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.030231 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.031586 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.031966 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.032036 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.034601 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.034603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.034604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.034612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.036889 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.037649 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.037651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.037651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.037717 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.037986 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.037987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.037988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.038133 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.038190 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.039210 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.039309 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":57,"threads_started":1}
I20260928 02:39:15.039868 16794 heartbeater.cc:347] Connected to a master server at 127.9.130.60:34557
I20260928 02:39:15.039976 16794 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:15.040153 16794 heartbeater.cc:511] Master 127.9.130.60:34557 requested a full tablet report, sending...
I20260928 02:39:15.040475 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.027563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:53983 (local address 127.9.130.61:40895)
0928 02:39:15.027611 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.027613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.027860 (+ 247us) server_negotiation.cc:407] Connection header received
0928 02:39:15.028713 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.028715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.028728 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.028758 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.029548 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.030306 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.031652 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.031974 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.032036 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.037386 (+ 5350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.037387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.037387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.037394 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.039137 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.039494 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.039495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.039496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.039558 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.039786 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.039788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.039789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.039986 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.040034 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.040289 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.040373 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:39:15.040679 16615 ts_manager.cc:194] Registered new tserver with Master: 9c06bd0797244592ae7e5370e71d7166 (127.9.130.1:43903)
I20260928 02:39:15.040684 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.027320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:40895 (local address 127.9.130.1:53983)
0928 02:39:15.027784 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.027793 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.028598 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.028882 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.028883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.029209 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:15.029440 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.029446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.030386 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.030386 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.031544 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.031546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.032195 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.032196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.032270 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.032874 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:15.032884 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:15.037287 (+ 4403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:15.039223 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.039224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.039224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.039414 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.039630 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.039631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.039632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.039699 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.040083 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:15.040083 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:15.040125 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.040196 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.040308 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":124,"thread_start_us":73,"threads_started":1}
I20260928 02:39:15.041152 16796 heartbeater.cc:347] Connected to a master server at 127.9.130.61:40895
I20260928 02:39:15.041204 16796 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:15.041278 16796 heartbeater.cc:511] Master 127.9.130.61:40895 requested a full tablet report, sending...
I20260928 02:39:15.041637 16422 ts_manager.cc:194] Registered new tserver with Master: 9c06bd0797244592ae7e5370e71d7166 (127.9.130.1:43903)
I20260928 02:39:15.042856 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:42421
--local_ip_for_outbound_sockets=127.9.130.2
--tserver_master_addrs=127.9.130.62:39455
--webserver_port=42359
--webserver_interface=127.9.130.2
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab
--principal=oryx/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.130.62:39455,127.9.130.61:40895
--tserver_master_addrs=127.9.130.62:39455,127.9.130.61:40895,127.9.130.60:34557 with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:39:15.043483 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.027282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.1:52647)
0928 02:39:15.027457 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.027500 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.028589 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.028879 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.028883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.029216 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:15.029440 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.029446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.030434 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.030435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.031593 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.031595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.032163 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.032164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.032270 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.032892 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:15.032896 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:15.034956 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:15.042497 (+ 7541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.042498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.042499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.042696 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.042895 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.042896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.042896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.042937 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.043240 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:15.043241 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:15.043279 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.043330 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.043402 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":46,"thread_start_us":93,"threads_started":1}
I20260928 02:39:15.043625 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.027274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:52647 (local address 127.9.130.62:39455)
0928 02:39:15.027323 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.027325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.027579 (+ 254us) server_negotiation.cc:407] Connection header received
0928 02:39:15.028722 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.028724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.028737 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.028791 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.029550 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.030310 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.031719 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.032045 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.032110 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.035047 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.035049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.035049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.035057 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.042313 (+ 7256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.042764 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.042766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.042766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.042826 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.043037 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.043039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.043039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.043166 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.043202 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.043394 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.043486 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:39:15.044389 16795 heartbeater.cc:347] Connected to a master server at 127.9.130.62:39455
I20260928 02:39:15.044445 16795 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:15.044505 16795 heartbeater.cc:511] Master 127.9.130.62:39455 requested a full tablet report, sending...
I20260928 02:39:15.044812 15823 ts_manager.cc:194] Re-registered known tserver with Master: 9c06bd0797244592ae7e5370e71d7166 (127.9.130.1:43903)
I20260928 02:39:15.045189 15823 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:52647
W20260928 02:39:15.127440 16802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:15.127628 16802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:15.127650 16802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:15.128801 16802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:15.128849 16802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:15.128875 16802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:15.130216 16802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab
--principal=oryx/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:42421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=42359
--tserver_master_addrs=127.9.130.62:39455,127.9.130.61:40895,127.9.130.60:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:15.130388 16802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:15.130553 16802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:15.135695 16808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:15.135696 16807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:15.135712 16810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:15.135815 16802 server_base.cc:1034] running on GCE node
I20260928 02:39:15.136143 16802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:15.136384 16802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:15.137542 16802 hybrid_clock.cc:648] HybridClock initialized: now 1790563155137530 us; error 37 us; skew 500 ppm
Sep 28 02:39:15 dist-test-slave-kkrh krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563155, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:15.140103 16802 init.cc:377] Logged in from keytab as oryx/127.9.130.2@KRBTEST.COM (short username oryx)
I20260928 02:39:15.140545 16802 webserver.cc:460] Webserver started at http://127.9.130.2:42359/ using document root <none> and password file <none>
I20260928 02:39:15.140764 16802 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:15.140844 16802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:15.142153 16802 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:15.142838 16817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:15.143034 16802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:15.143132 16802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "ab8c38454ade88f55ee9ea10527f6947"
server_key_iv: "1ef88bc8032fe75672899cc24b60ba4e"
server_key_version: "encryptionkey@0"
I20260928 02:39:15.143534 16802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:15.158217 16802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:15.160321 16802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:15.160499 16802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:15.160714 16802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:15.161037 16802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:15.161089 16802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:15.161123 16802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:15.161150 16802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:15.166801 16802 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:42421
I20260928 02:39:15.166880 16930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:42421 every 8 connection(s)
I20260928 02:39:15.167197 16802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:39:15.167853 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 16802
I20260928 02:39:15.168020 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 16155
I20260928 02:39:15.174077 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:35989
--local_ip_for_outbound_sockets=127.9.130.3
--tserver_master_addrs=127.9.130.62:39455
--webserver_port=34209
--webserver_interface=127.9.130.3
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab
--principal=oryx/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.130.62:39455,127.9.130.61:40895
--tserver_master_addrs=127.9.130.62:39455,127.9.130.61:40895,127.9.130.60:34557 with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:39:15 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563155, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
Sep 28 02:39:15 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563155, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.61@KRBTEST.COM
Sep 28 02:39:15 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563155, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.60@KRBTEST.COM
I20260928 02:39:15.180490 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.169948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:49605 (local address 127.9.130.61:40895)
0928 02:39:15.169992 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.169994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.170304 (+ 310us) server_negotiation.cc:407] Connection header received
0928 02:39:15.170402 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.170404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.170414 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.170444 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.170811 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.171511 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.173178 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.173463 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.173502 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.176996 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.176997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.176998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.177005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.178831 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.179330 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.179331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.179332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.179387 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.179670 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.179671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.179671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.179878 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.179935 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.180313 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.180401 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:39:15.180709 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.167927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:59717 (local address 127.9.130.60:34557)
0928 02:39:15.168004 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.168006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.169751 (+ 1745us) server_negotiation.cc:407] Connection header received
0928 02:39:15.169860 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.169861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.169877 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.169907 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.170633 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.171479 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.173070 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.173429 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.173476 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.177386 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.177388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.177388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.177395 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.179185 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.179605 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.179607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.179607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.179680 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.180056 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.180057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.180057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.180203 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.180261 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.180471 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.180573 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:39:15.180967 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.167905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:40709 (local address 127.9.130.62:39455)
0928 02:39:15.167951 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.167954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.168670 (+ 716us) server_negotiation.cc:407] Connection header received
0928 02:39:15.169266 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.169278 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.169290 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.169340 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.170605 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.171402 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.173072 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.173372 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.173413 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.177290 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.177292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.177293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.177301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.179705 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.180104 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.180105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.180105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.180161 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.180369 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.180370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.180371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.180494 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.180595 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.180785 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.180878 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:39:15.181351 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.169774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:40895 (local address 127.9.130.2:49605)
0928 02:39:15.170237 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.170247 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.170328 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.170504 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.170505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.170663 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:15.170717 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.170719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.171595 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.171595 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.173057 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.173058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.173574 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.173574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.173653 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.174520 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:15.174525 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:15.176900 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:15.178924 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.178930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.178932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.179236 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.179462 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.179463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.179464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.179594 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.179985 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:15.179991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:15.180155 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.180957 (+ 802us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.181254 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":622,"thread_start_us":80,"threads_started":1}
I20260928 02:39:15.181351 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.167917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.2:40709)
0928 02:39:15.168548 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.168590 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.169146 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.169406 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.169409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.170231 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:15.170503 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.170508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.171530 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.171530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.172967 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.172969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.173501 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.173501 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.173654 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.174334 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:15.174344 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:15.177173 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:15.179785 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.179786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.179786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.180026 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.180228 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.180229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.180229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.180296 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.180632 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:15.180633 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:15.180679 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.181098 (+ 419us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.181199 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":114,"thread_start_us":81,"threads_started":1}
I20260928 02:39:15.181337 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.168041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:34557 (local address 127.9.130.2:59717)
0928 02:39:15.169683 (+ 1642us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.169694 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.169779 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.169971 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.169972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.170221 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:15.170501 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.170508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.171575 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.171576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.172971 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.172974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.173571 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.173572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.173662 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.175588 (+ 1926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:15.175594 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:15.177268 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:15.179283 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.179284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.179285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.179517 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.179795 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.179796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.179797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.179966 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.180298 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:15.180298 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:15.180343 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.180852 (+ 509us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.181052 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1624,"mutex_wait_us":114,"thread_start_us":64,"threads_started":1}
I20260928 02:39:15.182080 16932 heartbeater.cc:347] Connected to a master server at 127.9.130.62:39455
I20260928 02:39:15.182190 16932 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:15.182255 16933 heartbeater.cc:347] Connected to a master server at 127.9.130.61:40895
I20260928 02:39:15.182310 16933 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:15.182387 16932 heartbeater.cc:511] Master 127.9.130.62:39455 requested a full tablet report, sending...
I20260928 02:39:15.182452 16933 heartbeater.cc:511] Master 127.9.130.61:40895 requested a full tablet report, sending...
I20260928 02:39:15.182757 15823 ts_manager.cc:194] Re-registered known tserver with Master: 8d8dbeca3d3a4b0795204c4fa4a2badc (127.9.130.2:42421)
I20260928 02:39:15.182919 16422 ts_manager.cc:194] Registered new tserver with Master: 8d8dbeca3d3a4b0795204c4fa4a2badc (127.9.130.2:42421)
I20260928 02:39:15.183133 16931 heartbeater.cc:347] Connected to a master server at 127.9.130.60:34557
I20260928 02:39:15.183175 16931 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:15.183225 16931 heartbeater.cc:511] Master 127.9.130.60:34557 requested a full tablet report, sending...
I20260928 02:39:15.183226 15823 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:40709
I20260928 02:39:15.183568 16615 ts_manager.cc:194] Registered new tserver with Master: 8d8dbeca3d3a4b0795204c4fa4a2badc (127.9.130.2:42421)
W20260928 02:39:15.259771 16938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:15.260052 16938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:15.260078 16938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:15.261272 16938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:15.261329 16938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:15.261358 16938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:15.262789 16938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab
--principal=oryx/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:35989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=34209
--tserver_master_addrs=127.9.130.62:39455,127.9.130.61:40895,127.9.130.60:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:15.262992 16938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:15.263181 16938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:15.268378 16946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:15.268378 16944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:15.268402 16943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:15.268625 16938 server_base.cc:1034] running on GCE node
I20260928 02:39:15.268822 16938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:15.269065 16938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:15.270242 16938 hybrid_clock.cc:648] HybridClock initialized: now 1790563155270230 us; error 24 us; skew 500 ppm
Sep 28 02:39:15 dist-test-slave-kkrh krb5kdc[15780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563155, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:15.272660 16938 init.cc:377] Logged in from keytab as oryx/127.9.130.3@KRBTEST.COM (short username oryx)
I20260928 02:39:15.273082 16938 webserver.cc:460] Webserver started at http://127.9.130.3:34209/ using document root <none> and password file <none>
I20260928 02:39:15.273312 16938 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:15.273391 16938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:15.274686 16938 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:15.275242 16953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:15.275406 16938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:15.275499 16938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "e542c7fd344b4e1aa71cb8b91a0593eb"
format_stamp: "Formatted at 2026-09-28 02:39:14 on dist-test-slave-kkrh"
server_key: "483de2c384b332dbdf12d3e27952ff95"
server_key_iv: "ffe1481ac6ed612151e86e3c4b46608a"
server_key_version: "encryptionkey@0"
I20260928 02:39:15.275882 16938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:15.311908 16938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:15.314518 16938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:15.314705 16938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:15.314965 16938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:15.315294 16938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:15.315351 16938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:15.315397 16938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:15.315428 16938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:15.321816 16938 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:35989
I20260928 02:39:15.321856 17066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:35989 every 8 connection(s)
I20260928 02:39:15.322243 16938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:39:15 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563155, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
Sep 28 02:39:15 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563155, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.60@KRBTEST.COM
I20260928 02:39:15.329780 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 16938
Sep 28 02:39:15 dist-test-slave-kkrh krb5kdc[15780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563155, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.61@KRBTEST.COM
I20260928 02:39:15.334396 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.323249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:49523 (local address 127.9.130.61:40895)
0928 02:39:15.323318 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.323321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.323503 (+ 182us) server_negotiation.cc:407] Connection header received
0928 02:39:15.324141 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.324143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.324155 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.324189 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.325065 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.325821 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.327294 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.327623 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.327697 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.330872 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.330873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.330873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.330880 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.332865 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.333399 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.333400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.333401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.333460 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.333700 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.333701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.333701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.333820 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.333863 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.334219 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.334295 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:39:15.334398 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.323236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:46537 (local address 127.9.130.62:39455)
0928 02:39:15.323319 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.323321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.323549 (+ 228us) server_negotiation.cc:407] Connection header received
0928 02:39:15.324129 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.324130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.324144 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.324205 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.325057 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.325600 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.326669 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.326936 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.327009 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.329596 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.329597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.329597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.329604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.332300 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.333347 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.333349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.333349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.333411 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.333707 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.333708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.333708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.333857 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.333902 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.334225 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.334307 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:39:15.334404 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.323052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:39061 (local address 127.9.130.60:34557)
0928 02:39:15.323103 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.323106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.323529 (+ 423us) server_negotiation.cc:407] Connection header received
0928 02:39:15.324142 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.324143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.324155 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.324189 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.325066 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.325816 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.326905 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.327172 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.327227 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.330824 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.330825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.330825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.330833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.332528 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.333655 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.333656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.333657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.333710 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.333907 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.333908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.333908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.334017 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.334058 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.334246 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.334324 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:39:15.334952 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.322969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:34557 (local address 127.9.130.3:39061)
0928 02:39:15.323344 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.323392 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.324016 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.324283 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.324287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.324734 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:15.324957 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.324962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.325914 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.325915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.326820 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.326822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.327249 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.327250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.327324 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.327952 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:15.327956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:15.330720 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:15.332617 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.332623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.332626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.332917 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.333784 (+ 867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.333785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.333785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.333834 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.334103 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:15.334103 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:15.334136 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.334517 (+ 381us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.334781 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":148,"thread_start_us":118,"threads_started":1}
I20260928 02:39:15.334954 17071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.322969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:40895 (local address 127.9.130.3:49523)
0928 02:39:15.323406 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.323415 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.324016 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.324283 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.324287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.324735 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:15.324957 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.324962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.325914 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.325915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.326832 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.326834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.327704 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.327705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.327770 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.328186 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:15.328191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:15.330720 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:15.332948 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.332949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.332950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.333304 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.333528 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.333528 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.333529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.333627 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.333909 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:15.333915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:15.334071 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.334516 (+ 445us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.334694 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":361,"thread_start_us":67,"threads_started":1}
I20260928 02:39:15.335538 17069 heartbeater.cc:347] Connected to a master server at 127.9.130.61:40895
I20260928 02:39:15.335637 17069 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:15.335752 17067 heartbeater.cc:347] Connected to a master server at 127.9.130.60:34557
I20260928 02:39:15.335866 17069 heartbeater.cc:511] Master 127.9.130.61:40895 requested a full tablet report, sending...
I20260928 02:39:15.335834 17067 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:15.335760 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.322969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.3:46537)
0928 02:39:15.323408 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.323418 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.324028 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.324592 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.324593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.324769 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:15.324956 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.324960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.325707 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.325708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.326563 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.326565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.327010 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.327010 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.327105 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.327614 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:15.327622 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:15.329500 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:15.332959 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.332960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.332961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.333249 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.333484 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:15.333485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:15.333485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:15.333631 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:15.333949 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:15.333949 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:15.334072 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.335414 (+ 1342us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.335518 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":7,"thread_start_us":106,"threads_started":1}
I20260928 02:39:15.335984 17067 heartbeater.cc:511] Master 127.9.130.60:34557 requested a full tablet report, sending...
I20260928 02:39:15.336362 16422 ts_manager.cc:194] Registered new tserver with Master: e542c7fd344b4e1aa71cb8b91a0593eb (127.9.130.3:35989)
I20260928 02:39:15.336618 16615 ts_manager.cc:194] Registered new tserver with Master: e542c7fd344b4e1aa71cb8b91a0593eb (127.9.130.3:35989)
I20260928 02:39:15.336841 17068 heartbeater.cc:347] Connected to a master server at 127.9.130.62:39455
I20260928 02:39:15.336895 17068 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:15.336966 17068 heartbeater.cc:511] Master 127.9.130.62:39455 requested a full tablet report, sending...
I20260928 02:39:15.337218 15823 ts_manager.cc:194] Re-registered known tserver with Master: e542c7fd344b4e1aa71cb8b91a0593eb (127.9.130.3:35989)
I20260928 02:39:15.337689 15823 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:46537
I20260928 02:39:15.341866 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.331410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.9.130.61:40895)
0928 02:39:15.332293 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.332295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.332303 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:15.332324 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.332324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.332335 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.333084 (+ 749us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.333119 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.334764 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.337161 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.337414 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.337479 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.338774 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.338775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.338776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.338782 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.340521 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.340994 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.340996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.340996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.341053 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.341265 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.341266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.341267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.341429 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.341534 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.341670 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.341748 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":56,"threads_started":1}
I20260928 02:39:15.342895 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.330888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.9.130.60:34557)
0928 02:39:15.331395 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.331397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.331566 (+ 169us) server_negotiation.cc:407] Connection header received
0928 02:39:15.331588 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.331589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.331601 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.331664 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.333609 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.334440 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.338248 (+ 3808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.338529 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.338570 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.340010 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.340011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.340012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.340018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.341681 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.342055 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.342056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.342056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.342108 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.342299 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.342300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.342301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.342455 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.342557 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.342701 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.342778 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":52,"threads_started":1}
I20260928 02:39:15.344269 17082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.331913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.9.130.62:39455)
0928 02:39:15.332608 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.332610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.335853 (+ 3243us) server_negotiation.cc:407] Connection header received
0928 02:39:15.335879 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.335880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.335896 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.335966 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.338251 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.338951 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.339782 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.340134 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.340210 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.341605 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.341606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.341607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.341615 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.343361 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.343648 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.343649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.343649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.343695 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.343872 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.343872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.343872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.343980 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.344056 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.344121 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.344181 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":71,"threads_started":1}
I20260928 02:39:15.345708 15823 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:39:15.347319 15823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:39:15.354856 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.351289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:35989 (local address 127.0.0.1:48886)
0928 02:39:15.351332 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.351345 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.351387 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.352013 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.352014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.352015 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:15.352058 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.352060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.353100 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.353101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.353949 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.353951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.354629 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.354630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.354687 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.354690 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.354746 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.354789 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 02:39:15.354868 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.351595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.9.130.3:35989)
0928 02:39:15.351778 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.351783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.351794 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:39:15.351869 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.351872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.351890 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.352013 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:15.352129 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.353019 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.354039 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.354530 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.354631 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.354711 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.354764 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.354798 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260928 02:39:15.355643 17001 tablet_service.cc:1467] Processing CreateTablet for tablet 7cecc7f8b363425e91cd54836e9bf118 (DEFAULT_TABLE table=test-table [id=8af37bf3e3ef4f87b97eb6dff4b6add8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:15.356022 17001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cecc7f8b363425e91cd54836e9bf118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:15.356747 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.351453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.9.130.1:43903)
0928 02:39:15.351628 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.351633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.352186 (+ 553us) server_negotiation.cc:407] Connection header received
0928 02:39:15.352251 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.352253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.352269 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.352350 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:15.352519 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.353752 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.355492 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.356263 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.356343 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.356418 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.356601 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.356650 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260928 02:39:15.357002 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.351291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:43903 (local address 127.0.0.1:42686)
0928 02:39:15.352108 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.352118 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.352174 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.352364 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.352371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.352373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:15.352416 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.352418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.353922 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.353923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.355418 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.355420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.356354 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.356355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.356419 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.356423 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.356866 (+ 443us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.356910 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":593,"thread_start_us":101,"threads_started":1}
I20260928 02:39:15.357179 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.351837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.9.130.2:42421)
0928 02:39:15.352294 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.352298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.352306 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:15.352366 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.352369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.352385 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.352476 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:15.352577 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.353748 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.355926 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.356659 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.356720 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.356799 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.357045 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.357092 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":71,"threads_started":1}
I20260928 02:39:15.357378 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.351403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:42421 (local address 127.0.0.1:35974)
0928 02:39:15.351546 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.351558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.351602 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.352472 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.352473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.352475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:15.352517 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.352519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.353855 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.353856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.355179 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.355182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.356841 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.356842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.356904 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.356907 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.356958 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.356990 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":33,"thread_start_us":92,"threads_started":1}
I20260928 02:39:15.357659 16728 tablet_service.cc:1467] Processing CreateTablet for tablet 7cecc7f8b363425e91cd54836e9bf118 (DEFAULT_TABLE table=test-table [id=8af37bf3e3ef4f87b97eb6dff4b6add8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:15.357983 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cecc7f8b363425e91cd54836e9bf118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:15.358019 16865 tablet_service.cc:1467] Processing CreateTablet for tablet 7cecc7f8b363425e91cd54836e9bf118 (DEFAULT_TABLE table=test-table [id=8af37bf3e3ef4f87b97eb6dff4b6add8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:15.358304 16865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cecc7f8b363425e91cd54836e9bf118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:15.360792 17093 tablet_bootstrap.cc:492] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb: Bootstrap starting.
I20260928 02:39:15.361984 17095 tablet_bootstrap.cc:492] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc: Bootstrap starting.
I20260928 02:39:15.362458 17093 tablet_bootstrap.cc:654] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:15.362530 17095 tablet_bootstrap.cc:654] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:15.362771 17093 log.cc:824] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:15.362960 17095 log.cc:824] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:15.363291 17094 tablet_bootstrap.cc:492] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166: Bootstrap starting.
I20260928 02:39:15.363579 17093 tablet_bootstrap.cc:492] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb: No bootstrap required, opened a new log
I20260928 02:39:15.363654 17093 ts_tablet_manager.cc:1397] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:39:15.363906 17094 tablet_bootstrap.cc:654] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:15.364162 17095 tablet_bootstrap.cc:492] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc: No bootstrap required, opened a new log
I20260928 02:39:15.364197 17094 log.cc:824] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:15.364272 17095 ts_tablet_manager.cc:1397] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:39:15.364923 17094 tablet_bootstrap.cc:492] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166: No bootstrap required, opened a new log
I20260928 02:39:15.364992 17094 ts_tablet_manager.cc:1397] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:39:15.366278 17093 raft_consensus.cc:348] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } }
I20260928 02:39:15.366438 17093 raft_consensus.cc:374] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:15.366463 17093 raft_consensus.cc:729] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e542c7fd344b4e1aa71cb8b91a0593eb, State: Initialized, Role: FOLLOWER
I20260928 02:39:15.366544 17093 consensus_queue.cc:260] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } }
I20260928 02:39:15.366779 17093 ts_tablet_manager.cc:1428] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:39:15.366930 17068 heartbeater.cc:503] Master 127.9.130.62:39455 was elected leader, sending a full tablet report...
I20260928 02:39:15.366817 17095 raft_consensus.cc:348] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } }
I20260928 02:39:15.366971 17095 raft_consensus.cc:374] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:15.367005 17095 raft_consensus.cc:729] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d8dbeca3d3a4b0795204c4fa4a2badc, State: Initialized, Role: FOLLOWER
I20260928 02:39:15.367111 17095 consensus_queue.cc:260] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } }
I20260928 02:39:15.367362 17095 ts_tablet_manager.cc:1428] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:39:15.367480 16932 heartbeater.cc:503] Master 127.9.130.62:39455 was elected leader, sending a full tablet report...
I20260928 02:39:15.368163 17094 raft_consensus.cc:348] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } }
I20260928 02:39:15.368300 17094 raft_consensus.cc:374] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:15.368332 17094 raft_consensus.cc:729] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c06bd0797244592ae7e5370e71d7166, State: Initialized, Role: FOLLOWER
I20260928 02:39:15.368453 17094 consensus_queue.cc:260] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } }
I20260928 02:39:15.368706 17094 ts_tablet_manager.cc:1428] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 02:39:15.368844 16795 heartbeater.cc:503] Master 127.9.130.62:39455 was elected leader, sending a full tablet report...
W20260928 02:39:15.419035 16934 tablet.cc:2369] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:39:15.464530 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.461977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.9.130.60:34557)
0928 02:39:15.462046 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.462048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.462071 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:39:15.462097 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.462098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.462118 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.462169 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:15.462304 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.462832 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.463744 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.464206 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.464267 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.464325 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.464435 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.464468 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:39:15.464562 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.461873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:34557 (local address 127.0.0.1:41818)
0928 02:39:15.461954 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.461973 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.462031 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.462189 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.462190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.462192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:15.462241 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.462242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.462927 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.462929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.463653 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.463655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.464297 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.464298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.464347 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.464350 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.464405 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.464448 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:39:15.464951 16625 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 02:39:15.465404 17102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 973bc1b3988a4e6ab2347fad49023e0b. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } }
I20260928 02:39:15.465519 17102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: This master's current role is: LEARNER
W20260928 02:39:15.528198 16797 tablet.cc:2369] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:39:15.573860 17070 tablet.cc:2369] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:39:15.614804 17099 raft_consensus.cc:482] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:39:15.614928 17099 raft_consensus.cc:504] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } }
I20260928 02:39:15.615360 17099 leader_election.cc:288] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c06bd0797244592ae7e5370e71d7166 (127.9.130.1:43903), 8d8dbeca3d3a4b0795204c4fa4a2badc (127.9.130.2:42421)
I20260928 02:39:15.618324 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.615629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:46579 (local address 127.9.130.1:43903)
0928 02:39:15.615676 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.615678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.615708 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:39:15.615790 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.615791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.615803 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.615894 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:15.616071 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.616523 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.617535 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.617920 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.618047 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.618056 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.618203 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.618239 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:39:15.618324 17071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.615551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:43903 (local address 127.9.130.3:46579)
0928 02:39:15.615649 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.615661 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.615697 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.615916 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.615917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.615920 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:15.615980 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.615981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.616602 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.616603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.617453 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.617456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.618046 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.618048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.618131 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.618136 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.618191 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.618230 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 02:39:15.618443 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.615609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:42421 (local address 127.9.130.3:56245)
0928 02:39:15.615677 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:15.615688 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:15.615728 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:15.615895 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:15.615896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:15.615900 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:15.615972 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.615974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.616651 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.616652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.617520 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:15.617523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.618129 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.618130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:15.618245 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.618249 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:39:15.618304 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:39:15.618340 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:39:15.618469 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.615655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:56245 (local address 127.9.130.2:42421)
0928 02:39:15.615719 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.615720 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.615731 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:39:15.615790 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.615791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.615803 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.615885 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:15.616040 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.616547 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.617619 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.618045 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.618122 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.618132 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.618344 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.618391 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:39:15.618646 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cecc7f8b363425e91cd54836e9bf118" candidate_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c06bd0797244592ae7e5370e71d7166" is_pre_election: true
I20260928 02:39:15.618813 16748 raft_consensus.cc:2454] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e542c7fd344b4e1aa71cb8b91a0593eb in term 0.
I20260928 02:39:15.618803 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cecc7f8b363425e91cd54836e9bf118" candidate_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" is_pre_election: true
I20260928 02:39:15.618937 16885 raft_consensus.cc:2454] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e542c7fd344b4e1aa71cb8b91a0593eb in term 0.
I20260928 02:39:15.619053 16956 leader_election.cc:302] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c06bd0797244592ae7e5370e71d7166, e542c7fd344b4e1aa71cb8b91a0593eb; no voters:
I20260928 02:39:15.619231 17099 raft_consensus.cc:2781] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:39:15.619282 17099 raft_consensus.cc:482] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:39:15.619297 17099 raft_consensus.cc:3037] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:15.620102 17099 raft_consensus.cc:504] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } }
I20260928 02:39:15.620237 17099 leader_election.cc:288] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [CANDIDATE]: Term 1 election: Requested vote from peers 9c06bd0797244592ae7e5370e71d7166 (127.9.130.1:43903), 8d8dbeca3d3a4b0795204c4fa4a2badc (127.9.130.2:42421)
I20260928 02:39:15.620394 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cecc7f8b363425e91cd54836e9bf118" candidate_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c06bd0797244592ae7e5370e71d7166"
I20260928 02:39:15.620536 16748 raft_consensus.cc:3037] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:15.621340 16748 raft_consensus.cc:2454] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e542c7fd344b4e1aa71cb8b91a0593eb in term 1.
I20260928 02:39:15.621434 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cecc7f8b363425e91cd54836e9bf118" candidate_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc"
I20260928 02:39:15.621501 16956 leader_election.cc:302] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c06bd0797244592ae7e5370e71d7166, e542c7fd344b4e1aa71cb8b91a0593eb; no voters:
I20260928 02:39:15.621544 16885 raft_consensus.cc:3037] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:15.621624 17099 raft_consensus.cc:2781] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:15.621798 17099 raft_consensus.cc:686] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [term 1 LEADER]: Becoming Leader. State: Replica: e542c7fd344b4e1aa71cb8b91a0593eb, State: Running, Role: LEADER
I20260928 02:39:15.621891 17099 consensus_queue.cc:237] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } }
I20260928 02:39:15.622541 16885 raft_consensus.cc:2454] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e542c7fd344b4e1aa71cb8b91a0593eb in term 1.
I20260928 02:39:15.622799 15825 catalog_manager.cc:5495] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb reported cstate change: term changed from 0 to 1, leader changed from <none> to e542c7fd344b4e1aa71cb8b91a0593eb (127.9.130.3). New cstate: current_term: 1 leader_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } health_report { overall_health: UNKNOWN } } }
I20260928 02:39:15.623430 16384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } }
I20260928 02:39:15.623525 16384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:15.624620 16384 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b: attempting to promote NON_VOTER 9c08a8244eaa49f5974db0eb4be56852 to VOTER
I20260928 02:39:15.624770 16384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } }
I20260928 02:39:15.625136 16625 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 LEARNER]: Refusing update from remote peer 973bc1b3988a4e6ab2347fad49023e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 02:39:15.625171 16432 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 FOLLOWER]: Refusing update from remote peer 973bc1b3988a4e6ab2347fad49023e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 02:39:15.625317 17108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 02:39:15.625411 16579 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 02:39:15.626027 16579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: NON_VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: true } } } pending_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } } }
I20260928 02:39:15.626052 16384 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 9c08a8244eaa49f5974db0eb4be56852 (127.9.130.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } } }
I20260928 02:39:15.626127 16579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:15.626122 16625 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 9c08a8244eaa49f5974db0eb4be56852 (127.9.130.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } } }
I20260928 02:39:15.626371 16432 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 9c08a8244eaa49f5974db0eb4be56852 (127.9.130.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } } }
I20260928 02:39:15.626910 16384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } } }
I20260928 02:39:15.626997 16384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:15.626941 16579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } } }
I20260928 02:39:15.627039 16579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:15.627104 17102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } } }
I20260928 02:39:15.627182 17102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:39:15.627250 17111 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "973bc1b3988a4e6ab2347fad49023e0b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } } }
I20260928 02:39:15.627348 17111 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:39:15.637210 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.634935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.9.130.3:35989)
0928 02:39:15.634980 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.634982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.634989 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:39:15.635054 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.635055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.635070 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.635129 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:39:15.635232 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.635675 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.636459 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.636654 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.636704 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.636851 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:39:15.636987 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:39:15.637054 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.637099 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.637117 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:39:15.638837 16885 raft_consensus.cc:1261] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 1 FOLLOWER]: Refusing update from remote peer e542c7fd344b4e1aa71cb8b91a0593eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:15.638844 16748 raft_consensus.cc:1261] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166 [term 1 FOLLOWER]: Refusing update from remote peer e542c7fd344b4e1aa71cb8b91a0593eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:15.639160 17106 consensus_queue.cc:1035] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:15.639298 17099 consensus_queue.cc:1035] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:15.642082 17114 mvcc.cc:205] Tried to move back new op lower bound from 7334146685494079488 to 7334146685427585024. Current Snapshot: MvccSnapshot[applied={T|T < 7334146685494079488}]
I20260928 02:39:15.642575 17116 mvcc.cc:205] Tried to move back new op lower bound from 7334146685494079488 to 7334146685427585024. Current Snapshot: MvccSnapshot[applied={T|T < 7334146685494079488}]
I20260928 02:39:15.643090 17113 mvcc.cc:205] Tried to move back new op lower bound from 7334146685494079488 to 7334146685427585024. Current Snapshot: MvccSnapshot[applied={T|T < 7334146685494079488}]
I20260928 02:39:15.645112 15825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:15.645182 15825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:15.646646 15825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 973bc1b3988a4e6ab2347fad49023e0b: Sending DeleteTablet for 3 replicas of tablet 7cecc7f8b363425e91cd54836e9bf118
I20260928 02:39:15.646952 16728 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cecc7f8b363425e91cd54836e9bf118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:15 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:42686
I20260928 02:39:15.646972 17001 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cecc7f8b363425e91cd54836e9bf118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:15 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:48886
I20260928 02:39:15.647164 17124 tablet_replica.cc:330] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb: stopping tablet replica
I20260928 02:39:15.647276 17124 raft_consensus.cc:2229] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:39:15.647320 16865 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cecc7f8b363425e91cd54836e9bf118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:15 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:35974
I20260928 02:39:15.647436 17124 raft_consensus.cc:2258] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:15.647579 17123 tablet_replica.cc:330] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166: stopping tablet replica
I20260928 02:39:15.647917 17123 raft_consensus.cc:2229] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:15.648031 17123 raft_consensus.cc:2258] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:15.648166 17125 tablet_replica.cc:330] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc: stopping tablet replica
I20260928 02:39:15.648188 17124 ts_tablet_manager.cc:1905] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:15.648262 17125 raft_consensus.cc:2229] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:15.648320 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 15793
I20260928 02:39:15.648355 17125 raft_consensus.cc:2258] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:15.649731 17123 ts_tablet_manager.cc:1905] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:15.650012 17124 ts_tablet_manager.cc:1918] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:15.650121 17124 log.cc:1196] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/7cecc7f8b363425e91cd54836e9bf118
I20260928 02:39:15.650450 17124 ts_tablet_manager.cc:1939] T 7cecc7f8b363425e91cd54836e9bf118 P e542c7fd344b4e1aa71cb8b91a0593eb: Deleting consensus metadata
I20260928 02:39:15.651665 17123 ts_tablet_manager.cc:1918] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:15.651734 17123 log.cc:1196] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-0/wal/wals/7cecc7f8b363425e91cd54836e9bf118
I20260928 02:39:15.652014 17123 ts_tablet_manager.cc:1939] T 7cecc7f8b363425e91cd54836e9bf118 P 9c06bd0797244592ae7e5370e71d7166: Deleting consensus metadata
I20260928 02:39:15.653403 17125 ts_tablet_manager.cc:1905] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:39:15.654838 16683 connection.cc:535] server connection from 127.0.0.1:42686 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:39:15.655037 16957 connection.cc:535] server connection from 127.0.0.1:48886 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:39:15.655130 17125 ts_tablet_manager.cc:1918] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:15.655190 17125 log.cc:1196] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/7cecc7f8b363425e91cd54836e9bf118
I20260928 02:39:15.655467 17125 ts_tablet_manager.cc:1939] T 7cecc7f8b363425e91cd54836e9bf118 P 8d8dbeca3d3a4b0795204c4fa4a2badc: Deleting consensus metadata
W20260928 02:39:15.655769 16820 connection.cc:441] server connection from 127.0.0.1:35974 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35974 (request call id 1) could send its response
I20260928 02:39:15.659873 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.647799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.9.130.60:34557)
0928 02:39:15.647860 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.647862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.647871 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:39:15.650901 (+ 3030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.650902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.650913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.650943 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.651885 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.653048 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.654360 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.654649 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.654691 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.656838 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.656840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.656840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.656847 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.658665 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.659061 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.659063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.659063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.659116 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.659307 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.659308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.659309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.659424 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.659503 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.659688 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.659763 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:39:15.664361 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.657068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.9.130.61:40895)
0928 02:39:15.657108 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.657110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.657117 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:39:15.657138 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.657139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.657151 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.657203 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.657834 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.658593 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.659606 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.659949 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.659999 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.661561 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.661562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.661563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.661570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.663142 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.663528 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.663529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.663530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.663588 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.663803 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.663804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.663804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.663948 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.664056 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.664160 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.664237 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:39:15.664805 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:15.657121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.9.130.60:34557)
0928 02:39:15.657225 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:15.657227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:15.657235 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:15.657253 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:15.657254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:15.657263 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:15.657321 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:15.658239 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.658918 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.659903 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:15.660189 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:15.660228 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:15.661848 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:15.661849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:15.661849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:15.661855 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:15.663406 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.663769 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.663770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.663770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.663843 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:15.664063 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:15.664064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:15.664064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:15.664181 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:15.664245 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:15.664644 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 02:39:15.664718 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 02:39:16.643405 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:16.643069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.2:55285)
0928 02:39:16.643262 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:16.643343 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":69,"threads_started":1}
W20260928 02:39:16.643620 16932 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:39455 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
I20260928 02:39:16.643888 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:16.643338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.1:40747)
0928 02:39:16.643645 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:16.643783 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":46,"threads_started":1}
I20260928 02:39:16.644007 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:16.643198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.3:40289)
0928 02:39:16.643903 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:16.643971 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":53,"threads_started":1}
W20260928 02:39:16.644210 17068 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:39455 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
W20260928 02:39:16.644270 16795 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:39455 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
I20260928 02:39:17.146947 17136 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 973bc1b3988a4e6ab2347fad49023e0b)
I20260928 02:39:17.147068 17136 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } }
I20260928 02:39:17.147286 17137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 973bc1b3988a4e6ab2347fad49023e0b)
I20260928 02:39:17.147426 17136 leader_election.cc:288] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 973bc1b3988a4e6ab2347fad49023e0b (127.9.130.62:39455), 9c08a8244eaa49f5974db0eb4be56852 (127.9.130.60:34557)
I20260928 02:39:17.147423 17137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } }
I20260928 02:39:17.147804 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.147560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47486)
0928 02:39:17.147706 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:17.147758 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":53,"threads_started":1}
W20260928 02:39:17.147964 16406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
I20260928 02:39:17.148051 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.147774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47500)
0928 02:39:17.147958 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:17.148004 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":67,"threads_started":1}
W20260928 02:39:17.148267 16599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
I20260928 02:39:17.148500 17137 leader_election.cc:288] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 973bc1b3988a4e6ab2347fad49023e0b (127.9.130.62:39455), cb1abc6d08c64bc9984156fc48da77be (127.9.130.61:40895)
I20260928 02:39:17.148514 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.148420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47510)
0928 02:39:17.148457 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:17.148488 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 02:39:17.148681 16406 leader_election.cc:334] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 973bc1b3988a4e6ab2347fad49023e0b (127.9.130.62:39455): Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
I20260928 02:39:17.148833 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.148738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47516)
0928 02:39:17.148776 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:17.148806 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 02:39:17.148917 16599 leader_election.cc:334] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 973bc1b3988a4e6ab2347fad49023e0b (127.9.130.62:39455): Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
I20260928 02:39:17.151050 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.147634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:34557 (local address 127.0.0.1:41850)
0928 02:39:17.147732 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:17.147743 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:17.147805 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:17.148050 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:17.148051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:17.148054 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:17.148149 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:17.148153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:17.149166 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:17.149167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:17.150262 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:17.150264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:17.150794 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:17.150795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:17.150856 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:17.150860 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:39:17.150941 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:39:17.150985 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":56,"threads_started":1}
I20260928 02:39:17.151047 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.147744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.9.130.60:34557)
0928 02:39:17.147914 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:17.147916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:17.147923 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:39:17.147949 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:17.147950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:17.147962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:17.148219 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:17.148256 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:17.149079 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:17.150348 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:17.150719 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:17.150795 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:17.150806 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:17.150948 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 02:39:17.150987 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":43,"threads_started":1}
I20260928 02:39:17.151372 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.147987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:40895 (local address 127.0.0.1:58252)
0928 02:39:17.148107 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:17.148116 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:17.148161 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:17.148965 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:17.148966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:17.148969 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:17.149011 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:17.149012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:17.149655 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:17.149657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:17.150578 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:17.150580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:17.151152 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:17.151153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:17.151214 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:17.151218 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:39:17.151265 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:39:17.151301 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":39,"threads_started":1}
I20260928 02:39:17.151425 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb1abc6d08c64bc9984156fc48da77be" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9c08a8244eaa49f5974db0eb4be56852" is_pre_election: true
I20260928 02:39:17.151535 16625 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb1abc6d08c64bc9984156fc48da77be in term 1.
I20260928 02:39:17.151567 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.148111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.9.130.61:40895)
0928 02:39:17.148541 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:17.148543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:17.148561 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:39:17.148585 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:17.148586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:17.148595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:17.149077 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:17.149104 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:17.149577 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:17.150676 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:17.151054 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:17.151147 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:17.151157 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:17.151345 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 02:39:17.151385 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":71,"threads_started":1}
I20260928 02:39:17.151706 16408 leader_election.cc:302] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c08a8244eaa49f5974db0eb4be56852, cb1abc6d08c64bc9984156fc48da77be; no voters: 973bc1b3988a4e6ab2347fad49023e0b
I20260928 02:39:17.151726 16432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c08a8244eaa49f5974db0eb4be56852" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cb1abc6d08c64bc9984156fc48da77be" is_pre_election: true
I20260928 02:39:17.151849 16432 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c08a8244eaa49f5974db0eb4be56852 in term 1.
I20260928 02:39:17.151890 17136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:39:17.151930 17136 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 973bc1b3988a4e6ab2347fad49023e0b)
I20260928 02:39:17.151960 17136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:39:17.151997 16600 leader_election.cc:302] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c08a8244eaa49f5974db0eb4be56852, cb1abc6d08c64bc9984156fc48da77be; no voters: 973bc1b3988a4e6ab2347fad49023e0b
I20260928 02:39:17.152113 17137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:39:17.152149 17137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 973bc1b3988a4e6ab2347fad49023e0b)
I20260928 02:39:17.152164 17137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:39:17.152693 17136 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } }
I20260928 02:39:17.152774 17137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } }
I20260928 02:39:17.152830 17136 leader_election.cc:288] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [CANDIDATE]: Term 2 election: Requested vote from peers 973bc1b3988a4e6ab2347fad49023e0b (127.9.130.62:39455), 9c08a8244eaa49f5974db0eb4be56852 (127.9.130.60:34557)
I20260928 02:39:17.152989 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb1abc6d08c64bc9984156fc48da77be" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9c08a8244eaa49f5974db0eb4be56852"
I20260928 02:39:17.153052 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.152894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47546)
0928 02:39:17.153003 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:17.153029 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 02:39:17.153057 16625 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cb1abc6d08c64bc9984156fc48da77be in current term 2: Already voted for candidate 9c08a8244eaa49f5974db0eb4be56852 in this term.
I20260928 02:39:17.153064 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.152973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47558)
0928 02:39:17.153010 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:17.153033 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 02:39:17.153189 16432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c08a8244eaa49f5974db0eb4be56852" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cb1abc6d08c64bc9984156fc48da77be"
I20260928 02:39:17.153256 16432 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c08a8244eaa49f5974db0eb4be56852 in current term 2: Already voted for candidate cb1abc6d08c64bc9984156fc48da77be in this term.
I20260928 02:39:17.153343 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.153255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47570)
0928 02:39:17.153291 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:17.153314 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 02:39:17.153388 16406 leader_election.cc:334] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 973bc1b3988a4e6ab2347fad49023e0b (127.9.130.62:39455): Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
I20260928 02:39:17.153276 17137 leader_election.cc:288] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [CANDIDATE]: Term 2 election: Requested vote from peers 973bc1b3988a4e6ab2347fad49023e0b (127.9.130.62:39455), cb1abc6d08c64bc9984156fc48da77be (127.9.130.61:40895)
I20260928 02:39:17.153414 16406 leader_election.cc:302] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb1abc6d08c64bc9984156fc48da77be; no voters: 973bc1b3988a4e6ab2347fad49023e0b, 9c08a8244eaa49f5974db0eb4be56852
I20260928 02:39:17.153470 17136 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 02:39:17.153555 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.153477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47582)
0928 02:39:17.153508 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:17.153535 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 02:39:17.153627 16599 leader_election.cc:334] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 973bc1b3988a4e6ab2347fad49023e0b (127.9.130.62:39455): Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
I20260928 02:39:17.153676 16599 leader_election.cc:302] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c08a8244eaa49f5974db0eb4be56852; no voters: 973bc1b3988a4e6ab2347fad49023e0b, cb1abc6d08c64bc9984156fc48da77be
I20260928 02:39:17.153782 17137 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 02:39:17.644483 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.644180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.2:42817)
0928 02:39:17.644376 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:17.644421 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260928 02:39:17.644788 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.644576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.1:44605)
0928 02:39:17.644702 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:17.644740 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":49,"threads_started":1}
I20260928 02:39:17.644968 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:17.644796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.3:44167)
0928 02:39:17.644889 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:17.644919 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":45,"threads_started":1}
I20260928 02:39:18.645619 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:18.644922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.2:47145)
0928 02:39:18.645521 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:18.645563 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":63,"threads_started":1}
I20260928 02:39:18.645706 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:18.645403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.1:38191)
0928 02:39:18.645608 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:18.645647 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":50,"threads_started":1}
I20260928 02:39:18.645869 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:18.645640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.3:53989)
0928 02:39:18.645783 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:18.645820 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":45,"threads_started":1}
I20260928 02:39:19.120311 17152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:39:19.120428 17152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } }
I20260928 02:39:19.120599 17152 leader_election.cc:288] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 973bc1b3988a4e6ab2347fad49023e0b (127.9.130.62:39455), cb1abc6d08c64bc9984156fc48da77be (127.9.130.61:40895)
I20260928 02:39:19.120864 16432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c08a8244eaa49f5974db0eb4be56852" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cb1abc6d08c64bc9984156fc48da77be" is_pre_election: true
I20260928 02:39:19.120949 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.120727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47838)
0928 02:39:19.120858 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.120906 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":55,"threads_started":1}
I20260928 02:39:19.120980 16432 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c08a8244eaa49f5974db0eb4be56852 in term 2.
I20260928 02:39:19.121143 16600 leader_election.cc:302] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c08a8244eaa49f5974db0eb4be56852, cb1abc6d08c64bc9984156fc48da77be; no voters:
I20260928 02:39:19.121260 17152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 02:39:19.121345 17152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:39:19.121368 17152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:39:19.121408 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.121275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47854)
0928 02:39:19.121336 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.121366 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 02:39:19.121474 16599 leader_election.cc:334] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 973bc1b3988a4e6ab2347fad49023e0b (127.9.130.62:39455): Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
I20260928 02:39:19.122162 17152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } }
I20260928 02:39:19.122298 17152 leader_election.cc:288] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [CANDIDATE]: Term 3 election: Requested vote from peers 973bc1b3988a4e6ab2347fad49023e0b (127.9.130.62:39455), cb1abc6d08c64bc9984156fc48da77be (127.9.130.61:40895)
I20260928 02:39:19.122479 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.122396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47870)
0928 02:39:19.122429 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.122454 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 02:39:19.122450 16432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c08a8244eaa49f5974db0eb4be56852" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cb1abc6d08c64bc9984156fc48da77be"
I20260928 02:39:19.122545 16432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:39:19.122763 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.122678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47886)
0928 02:39:19.122721 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.122745 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:39:19.122840 16599 leader_election.cc:334] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 973bc1b3988a4e6ab2347fad49023e0b (127.9.130.62:39455): Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
I20260928 02:39:19.123189 16432 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c08a8244eaa49f5974db0eb4be56852 in term 3.
I20260928 02:39:19.123350 16600 leader_election.cc:302] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9c08a8244eaa49f5974db0eb4be56852, cb1abc6d08c64bc9984156fc48da77be; no voters: 973bc1b3988a4e6ab2347fad49023e0b
I20260928 02:39:19.123426 17152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:39:19.123517 17152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [term 3 LEADER]: Becoming Leader. State: Replica: 9c08a8244eaa49f5974db0eb4be56852, State: Running, Role: LEADER
I20260928 02:39:19.123602 17152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } }
I20260928 02:39:19.124040 17155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c08a8244eaa49f5974db0eb4be56852. Latest consensus state: current_term: 3 leader_uuid: "9c08a8244eaa49f5974db0eb4be56852" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } } }
I20260928 02:39:19.124150 17155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:19.124346 17157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:19.124766 17157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:19.124897 17157 catalog_manager.cc:1259] Loaded cluster ID: a65f718419f741969313ea95bf098fd5
I20260928 02:39:19.124930 17157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:39:19.125108 17157 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:19.125198 17157 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: Loaded TSK: 0
I20260928 02:39:19.125340 17157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:39:19.162726 16613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:39:19.164444 16613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:39:19.165390 16432 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [term 3 FOLLOWER]: Refusing update from remote peer 9c08a8244eaa49f5974db0eb4be56852: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 02:39:19.165421 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.165259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47910)
0928 02:39:19.165336 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.165383 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 02:39:19.165613 17155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 02:39:19.165747 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.165632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.0.0.1:47916)
0928 02:39:19.165687 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.165711 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 02:39:19.165818 16599 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 -> Peer 973bc1b3988a4e6ab2347fad49023e0b (127.9.130.62:39455): Couldn't send request to peer 973bc1b3988a4e6ab2347fad49023e0b. Status: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:39:19.166172 17159 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c08a8244eaa49f5974db0eb4be56852. Latest consensus state: current_term: 3 leader_uuid: "9c08a8244eaa49f5974db0eb4be56852" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } } }
I20260928 02:39:19.166240 17159 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:39:19.166720 17155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9c08a8244eaa49f5974db0eb4be56852" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } } }
I20260928 02:39:19.166812 17155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:19.166962 17159 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9c08a8244eaa49f5974db0eb4be56852" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "973bc1b3988a4e6ab2347fad49023e0b" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 39455 } } peers { permanent_uuid: "cb1abc6d08c64bc9984156fc48da77be" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 40895 } attrs { promote: false } } peers { permanent_uuid: "9c08a8244eaa49f5974db0eb4be56852" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 34557 } attrs { promote: false } } }
I20260928 02:39:19.167083 17159 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1abc6d08c64bc9984156fc48da77be [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:39:19.171896 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.168611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:35989 (local address 127.0.0.1:48900)
0928 02:39:19.168705 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.168718 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:19.168781 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:19.169386 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:19.169386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:19.169388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:19.169419 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:19.169420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.170231 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.170231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:19.170994 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:19.170996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.171645 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.171646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:19.171705 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:19.171708 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:19.171762 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:39:19.171802 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 02:39:19.171898 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.168740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.9.130.3:35989)
0928 02:39:19.169284 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:19.169287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:19.169294 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:39:19.169318 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:19.169319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:19.169328 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:19.169386 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:19.169473 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.170159 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.171089 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.171547 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.171644 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:19.171657 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:19.171772 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 02:39:19.171811 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":65,"threads_started":1}
I20260928 02:39:19.172232 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.168703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:42421 (local address 127.0.0.1:35980)
0928 02:39:19.168983 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.168993 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:19.169045 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:19.169216 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:19.169217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:19.169219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:19.169268 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:19.169270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.169975 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.169976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:19.171227 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:19.171229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.172022 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.172023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:19.172075 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:19.172082 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:39:19.172138 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:39:19.172166 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260928 02:39:19.172276 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.168873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.9.130.2:42421)
0928 02:39:19.168962 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:19.168964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:19.169100 (+ 136us) server_negotiation.cc:407] Connection header received
0928 02:39:19.169126 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:19.169127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:19.169136 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:19.169208 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:19.169358 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.169897 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.171332 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.171917 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.171984 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:19.171997 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:19.172154 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:39:19.172200 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":63,"threads_started":1}
I20260928 02:39:19.172438 17001 tablet_service.cc:1467] Processing CreateTablet for tablet 25b40abddba04c4dba016ab9003f558c (DEFAULT_TABLE table=test-table [id=828bbc073bfc48b6ad59ecb270458644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:19.172479 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.168866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.9.130.1:43903)
0928 02:39:19.168980 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:19.168982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:19.168991 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:39:19.169019 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:19.169020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:19.169031 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:19.169254 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:19.169293 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.170076 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.171329 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.171917 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.172319 (+ 402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:19.172337 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:19.172359 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 02:39:19.172409 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260928 02:39:19.172598 17001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b40abddba04c4dba016ab9003f558c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:19.172242 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.168611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:43903 (local address 127.0.0.1:42698)
0928 02:39:19.168753 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.168763 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:19.168818 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:19.169124 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:19.169125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:19.169127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:19.169188 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:19.169190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.170162 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.170162 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:19.171227 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:19.171229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.172014 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.172015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:19.172075 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:19.172082 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:39:19.172112 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 02:39:19.172156 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":29,"thread_start_us":69,"threads_started":1}
I20260928 02:39:19.172941 16865 tablet_service.cc:1467] Processing CreateTablet for tablet 25b40abddba04c4dba016ab9003f558c (DEFAULT_TABLE table=test-table [id=828bbc073bfc48b6ad59ecb270458644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:19.172958 16728 tablet_service.cc:1467] Processing CreateTablet for tablet 25b40abddba04c4dba016ab9003f558c (DEFAULT_TABLE table=test-table [id=828bbc073bfc48b6ad59ecb270458644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:19.173051 16865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b40abddba04c4dba016ab9003f558c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:19.173079 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b40abddba04c4dba016ab9003f558c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:19.174067 17170 tablet_bootstrap.cc:492] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166: Bootstrap starting.
I20260928 02:39:19.174386 17170 tablet_bootstrap.cc:654] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:19.174490 17172 tablet_bootstrap.cc:492] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc: Bootstrap starting.
I20260928 02:39:19.174885 17172 tablet_bootstrap.cc:654] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:19.174887 17171 tablet_bootstrap.cc:492] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb: Bootstrap starting.
I20260928 02:39:19.175024 17170 tablet_bootstrap.cc:492] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166: No bootstrap required, opened a new log
I20260928 02:39:19.175096 17170 ts_tablet_manager.cc:1397] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:19.175243 17171 tablet_bootstrap.cc:654] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:19.175251 17170 raft_consensus.cc:348] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } }
I20260928 02:39:19.175338 17170 raft_consensus.cc:374] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:19.175366 17170 raft_consensus.cc:729] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c06bd0797244592ae7e5370e71d7166, State: Initialized, Role: FOLLOWER
I20260928 02:39:19.175416 17170 consensus_queue.cc:260] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } }
I20260928 02:39:19.175573 17170 ts_tablet_manager.cc:1428] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:19.175940 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.175721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.1:32841)
0928 02:39:19.175865 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.175900 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260928 02:39:19.176108 17172 tablet_bootstrap.cc:492] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc: No bootstrap required, opened a new log
I20260928 02:39:19.176151 16794 heartbeater.cc:503] Master 127.9.130.60:34557 was elected leader, sending a full tablet report...
I20260928 02:39:19.176163 17172 ts_tablet_manager.cc:1397] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:39:19.176281 17172 raft_consensus.cc:348] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } }
I20260928 02:39:19.176362 17172 raft_consensus.cc:374] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:19.176389 17172 raft_consensus.cc:729] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d8dbeca3d3a4b0795204c4fa4a2badc, State: Initialized, Role: FOLLOWER
I20260928 02:39:19.176399 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.176294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.1:53821)
0928 02:39:19.176347 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.176373 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 02:39:19.176427 17172 consensus_queue.cc:260] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } }
I20260928 02:39:19.176606 17172 ts_tablet_manager.cc:1428] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:19.176723 17171 tablet_bootstrap.cc:492] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb: No bootstrap required, opened a new log
I20260928 02:39:19.176779 17171 ts_tablet_manager.cc:1397] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:39:19.177120 17171 raft_consensus.cc:348] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } }
I20260928 02:39:19.177189 17171 raft_consensus.cc:374] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:19.177202 17171 raft_consensus.cc:729] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e542c7fd344b4e1aa71cb8b91a0593eb, State: Initialized, Role: FOLLOWER
I20260928 02:39:19.177210 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.176890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.2:46935)
0928 02:39:19.177146 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.177183 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260928 02:39:19.177234 17171 consensus_queue.cc:260] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } }
I20260928 02:39:19.177531 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.177437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.2:49515)
0928 02:39:19.177481 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.177510 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:39:19.177604 16931 heartbeater.cc:503] Master 127.9.130.60:34557 was elected leader, sending a full tablet report...
I20260928 02:39:19.177712 17171 ts_tablet_manager.cc:1428] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:39:19.178174 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.177903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.3:39077)
0928 02:39:19.178094 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.178122 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260928 02:39:19.178272 17067 heartbeater.cc:503] Master 127.9.130.60:34557 was elected leader, sending a full tablet report...
I20260928 02:39:19.178721 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.178612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.3:56251)
0928 02:39:19.178668 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.178697 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:39:19.229056 17176 raft_consensus.cc:482] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:39:19.229146 17176 raft_consensus.cc:504] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } }
I20260928 02:39:19.229435 17176 leader_election.cc:288] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e542c7fd344b4e1aa71cb8b91a0593eb (127.9.130.3:35989), 8d8dbeca3d3a4b0795204c4fa4a2badc (127.9.130.2:42421)
I20260928 02:39:19.232968 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.229908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:40923 (local address 127.9.130.2:42421)
0928 02:39:19.229972 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:19.229974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:19.230078 (+ 104us) server_negotiation.cc:407] Connection header received
0928 02:39:19.230135 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:19.230136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:19.230147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:19.230206 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:19.230355 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.230890 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.232193 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.232562 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.232615 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:19.232624 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:19.232841 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 02:39:19.232882 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:39:19.232964 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.229805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:42421 (local address 127.9.130.1:40923)
0928 02:39:19.230006 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.230017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:19.230077 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:19.230227 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:19.230228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:19.230229 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:19.230290 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:19.230291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.230969 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.230970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:19.232090 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:19.232093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.232637 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.232639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:19.232717 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:19.232722 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:39:19.232827 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:39:19.232868 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":131,"threads_started":1}
I20260928 02:39:19.233089 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.229717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:46661 (local address 127.9.130.3:35989)
0928 02:39:19.229863 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:19.229865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:19.229875 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:39:19.229910 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:19.229911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:19.229921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:19.230022 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:19.230228 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.230975 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.232158 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.232697 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.232776 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:19.232790 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:19.232972 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 02:39:19.233018 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260928 02:39:19.233184 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.229620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:35989 (local address 127.9.130.1:46661)
0928 02:39:19.229680 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.229693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:19.229746 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:19.230045 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:19.230046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:19.230048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:19.230141 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:19.230143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.231065 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.231066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:19.232074 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:19.232077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.232813 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.232814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:19.232887 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:19.232890 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:19.232946 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:39:19.232982 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":1}
I20260928 02:39:19.233309 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b40abddba04c4dba016ab9003f558c" candidate_uuid: "9c06bd0797244592ae7e5370e71d7166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" is_pre_election: true
I20260928 02:39:19.233405 16885 raft_consensus.cc:2454] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c06bd0797244592ae7e5370e71d7166 in term 0.
I20260928 02:39:19.233434 17021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b40abddba04c4dba016ab9003f558c" candidate_uuid: "9c06bd0797244592ae7e5370e71d7166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" is_pre_election: true
I20260928 02:39:19.233557 17021 raft_consensus.cc:2454] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c06bd0797244592ae7e5370e71d7166 in term 0.
I20260928 02:39:19.233621 16682 leader_election.cc:302] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d8dbeca3d3a4b0795204c4fa4a2badc, 9c06bd0797244592ae7e5370e71d7166; no voters:
I20260928 02:39:19.233757 17176 raft_consensus.cc:2781] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:39:19.233807 17176 raft_consensus.cc:482] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:39:19.233830 17176 raft_consensus.cc:3037] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:19.234493 17176 raft_consensus.cc:504] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } }
I20260928 02:39:19.234625 17176 leader_election.cc:288] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [CANDIDATE]: Term 1 election: Requested vote from peers e542c7fd344b4e1aa71cb8b91a0593eb (127.9.130.3:35989), 8d8dbeca3d3a4b0795204c4fa4a2badc (127.9.130.2:42421)
I20260928 02:39:19.234834 17021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b40abddba04c4dba016ab9003f558c" candidate_uuid: "9c06bd0797244592ae7e5370e71d7166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb"
I20260928 02:39:19.234916 17021 raft_consensus.cc:3037] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:19.234897 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b40abddba04c4dba016ab9003f558c" candidate_uuid: "9c06bd0797244592ae7e5370e71d7166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc"
I20260928 02:39:19.235003 16885 raft_consensus.cc:3037] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:19.235594 16885 raft_consensus.cc:2454] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c06bd0797244592ae7e5370e71d7166 in term 1.
I20260928 02:39:19.235616 17021 raft_consensus.cc:2454] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c06bd0797244592ae7e5370e71d7166 in term 1.
I20260928 02:39:19.235798 16682 leader_election.cc:302] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d8dbeca3d3a4b0795204c4fa4a2badc, 9c06bd0797244592ae7e5370e71d7166; no voters:
I20260928 02:39:19.235921 17176 raft_consensus.cc:2781] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:19.236070 17176 raft_consensus.cc:686] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [term 1 LEADER]: Becoming Leader. State: Replica: 9c06bd0797244592ae7e5370e71d7166, State: Running, Role: LEADER
I20260928 02:39:19.236128 17176 consensus_queue.cc:237] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } }
I20260928 02:39:19.236433 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.236335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.1:50279)
0928 02:39:19.236368 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.236406 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 02:39:19.237017 16614 catalog_manager.cc:5495] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c06bd0797244592ae7e5370e71d7166 (127.9.130.1). New cstate: current_term: 1 leader_uuid: "9c06bd0797244592ae7e5370e71d7166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c06bd0797244592ae7e5370e71d7166" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 43903 } health_report { overall_health: HEALTHY } } }
I20260928 02:39:19.246805 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.244215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.9.130.1:43903)
0928 02:39:19.244286 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:19.244288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:19.244296 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:19.244328 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:19.244329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:19.244345 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:19.244406 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:39:19.244537 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.245004 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.245748 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:19.246043 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:19.246128 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:19.246341 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:39:19.246473 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:39:19.246548 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:19.246709 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:39:19.246730 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:39:19.249014 16885 raft_consensus.cc:1261] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 1 FOLLOWER]: Refusing update from remote peer 9c06bd0797244592ae7e5370e71d7166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:19.249094 17021 raft_consensus.cc:1261] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb [term 1 FOLLOWER]: Refusing update from remote peer 9c06bd0797244592ae7e5370e71d7166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:19.249356 17176 consensus_queue.cc:1035] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e542c7fd344b4e1aa71cb8b91a0593eb" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 35989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:19.249539 17183 consensus_queue.cc:1035] T 25b40abddba04c4dba016ab9003f558c P 9c06bd0797244592ae7e5370e71d7166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d8dbeca3d3a4b0795204c4fa4a2badc" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:19.250363 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.250123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.3:51309)
0928 02:39:19.250293 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.250327 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260928 02:39:19.250427 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.250329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.2:55545)
0928 02:39:19.250370 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.250395 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 02:39:19.251287 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.251086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.1:43255)
0928 02:39:19.251232 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.251261 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260928 02:39:19.251746 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.251637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.2:60437)
0928 02:39:19.251688 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.251714 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:39:19.251966 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.251880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.1:46239)
0928 02:39:19.251916 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.251940 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 02:39:19.252668 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.252571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:39455 (local address 127.9.130.3:42659)
0928 02:39:19.252612 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.252636 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 02:39:19.253782 16613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:19.253870 16613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:19.255064 16613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c08a8244eaa49f5974db0eb4be56852: Sending DeleteTablet for 3 replicas of tablet 25b40abddba04c4dba016ab9003f558c
I20260928 02:39:19.255371 16728 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b40abddba04c4dba016ab9003f558c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:19 UTC) from {username='oryx', principal='oryx/127.9.130.60@KRBTEST.COM'} at 127.0.0.1:42698
I20260928 02:39:19.255398 17001 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b40abddba04c4dba016ab9003f558c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:19 UTC) from {username='oryx', principal='oryx/127.9.130.60@KRBTEST.COM'} at 127.0.0.1:48900
I20260928 02:39:19.255513 16865 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b40abddba04c4dba016ab9003f558c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:19 UTC) from {username='oryx', principal='oryx/127.9.130.60@KRBTEST.COM'} at 127.0.0.1:35980
I20260928 02:39:19.255599 17197 tablet_replica.cc:330] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb: stopping tablet replica
I20260928 02:39:19.255652 17197 raft_consensus.cc:2229] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:19.255525 16864 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b40abddba04c4dba016ab9003f558c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:19 UTC) from {username='oryx', principal='oryx/127.9.130.60@KRBTEST.COM'} at 127.0.0.1:35980
I20260928 02:39:19.255692 17197 raft_consensus.cc:2258] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:19.255698 17198 tablet_replica.cc:330] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc: stopping tablet replica
I20260928 02:39:19.255766 17198 raft_consensus.cc:2229] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:19.255877 17198 raft_consensus.cc:2258] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:19.256016 17197 ts_tablet_manager.cc:1905] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:19.256270 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 16569
I20260928 02:39:19.259093 17197 ts_tablet_manager.cc:1918] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:19.259179 17197 log.cc:1196] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/25b40abddba04c4dba016ab9003f558c
I20260928 02:39:19.259155 17198 ts_tablet_manager.cc:1905] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:19.259490 17197 ts_tablet_manager.cc:1939] T 25b40abddba04c4dba016ab9003f558c P e542c7fd344b4e1aa71cb8b91a0593eb: Deleting consensus metadata
I20260928 02:39:19.259855 16600 catalog_manager.cc:4841] TS e542c7fd344b4e1aa71cb8b91a0593eb (127.9.130.3:35989): tablet 25b40abddba04c4dba016ab9003f558c (table test-table [id=828bbc073bfc48b6ad59ecb270458644]) successfully deleted
I20260928 02:39:19.260560 17198 ts_tablet_manager.cc:1918] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:19.260612 17198 log.cc:1196] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/25b40abddba04c4dba016ab9003f558c
I20260928 02:39:19.260828 17198 ts_tablet_manager.cc:1939] T 25b40abddba04c4dba016ab9003f558c P 8d8dbeca3d3a4b0795204c4fa4a2badc: Deleting consensus metadata
I20260928 02:39:19.261271 16600 catalog_manager.cc:4841] TS 8d8dbeca3d3a4b0795204c4fa4a2badc (127.9.130.2:42421): tablet 25b40abddba04c4dba016ab9003f558c (table test-table [id=828bbc073bfc48b6ad59ecb270458644]) successfully deleted
W20260928 02:39:19.261406 16600 catalog_manager.cc:4808] TS 8d8dbeca3d3a4b0795204c4fa4a2badc (127.9.130.2:42421): delete failed for tablet 25b40abddba04c4dba016ab9003f558c because the tablet was not found. No further retry: Not found: Tablet not found: 25b40abddba04c4dba016ab9003f558c
I20260928 02:39:19.263680 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 16802
I20260928 02:39:19.263700 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:19.263587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:43903 (local address 127.0.0.1:42722)
0928 02:39:19.263635 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:19.263662 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:43903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:39:19.263875 16601 catalog_manager.cc:4584] TS 9c06bd0797244592ae7e5370e71d7166 (127.9.130.1:43903): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 25b40abddba04c4dba016ab9003f558c: Network error: Client connection negotiation failed: client connection to 127.9.130.1:43903: connect: Connection refused (error 111)
I20260928 02:39:19.268666 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 16938
I20260928 02:39:19.275388 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 16289
I20260928 02:39:19.280023 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 16476
2026-09-28T02:39:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6752 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:39:19 dist-test-slave-kkrh krb5kdc[17201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:39:19 dist-test-slave-kkrh krb5kdc[17201](info): set up 2 sockets
Sep 28 02:39:19 dist-test-slave-kkrh krb5kdc[17201](info): commencing operation
krb5kdc: starting...
W20260928 02:39:20.400393 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:39:20 dist-test-slave-kkrh krb5kdc[17201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:39:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.62.keytab.
Entry for principal oryx/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.62.keytab.
I20260928 02:39:20.523195 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:38137
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.62.keytab
--principal=oryx/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:38137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:20.605362 17214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:20.605558 17214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:20.605580 17214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:20.606712 17214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:20.606750 17214 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:39:20.606763 17214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:20.606776 17214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:20.606827 17214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:20.606845 17214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:20.608417 17214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:38137
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.62.keytab
--principal=oryx/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:38137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:20.608660 17214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:20.608875 17214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:20.613638 17220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:20.613646 17219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:20.613870 17214 server_base.cc:1034] running on GCE node
W20260928 02:39:20.613675 17222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:20.614199 17214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:20.614465 17214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:20.615633 17214 hybrid_clock.cc:648] HybridClock initialized: now 1790563160615607 us; error 37 us; skew 500 ppm
Sep 28 02:39:20 dist-test-slave-kkrh krb5kdc[17201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563160, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:20.618039 17214 init.cc:377] Logged in from keytab as oryx/127.9.130.62@KRBTEST.COM (short username oryx)
I20260928 02:39:20.618439 17214 webserver.cc:460] Webserver started at http://127.9.130.62:40383/ using document root <none> and password file <none>
I20260928 02:39:20.618619 17214 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:20.618659 17214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:20.618743 17214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:20.619681 17214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "9937968dbb6b4fda88022d40089c2d62"
format_stamp: "Formatted at 2026-09-28 02:39:20 on dist-test-slave-kkrh"
server_key: "ebd796d5ebbcdcc80596f89992bc1f89"
server_key_iv: "a83a86efa0d180714308d5df3cc76532"
server_key_version: "encryptionkey@0"
I20260928 02:39:20.620076 17214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "9937968dbb6b4fda88022d40089c2d62"
format_stamp: "Formatted at 2026-09-28 02:39:20 on dist-test-slave-kkrh"
server_key: "ebd796d5ebbcdcc80596f89992bc1f89"
server_key_iv: "a83a86efa0d180714308d5df3cc76532"
server_key_version: "encryptionkey@0"
I20260928 02:39:20.621387 17214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:20.622087 17229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:20.622293 17214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:20.622375 17214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "9937968dbb6b4fda88022d40089c2d62"
format_stamp: "Formatted at 2026-09-28 02:39:20 on dist-test-slave-kkrh"
server_key: "ebd796d5ebbcdcc80596f89992bc1f89"
server_key_iv: "a83a86efa0d180714308d5df3cc76532"
server_key_version: "encryptionkey@0"
I20260928 02:39:20.622426 17214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:20.633560 17214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:20.635797 17214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:20.635978 17214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:20.640380 17214 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:38137
I20260928 02:39:20.640458 17291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:38137 every 8 connection(s)
I20260928 02:39:20.640805 17214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:20.641486 17292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:20.643664 17292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62: Bootstrap starting.
I20260928 02:39:20.644300 17292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:20.644586 17292 log.cc:824] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:20.645246 17292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62: No bootstrap required, opened a new log
I20260928 02:39:20.647197 17292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9937968dbb6b4fda88022d40089c2d62" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38137 } }
I20260928 02:39:20.647351 17292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:20.647395 17292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9937968dbb6b4fda88022d40089c2d62, State: Initialized, Role: FOLLOWER
I20260928 02:39:20.647529 17292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9937968dbb6b4fda88022d40089c2d62" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38137 } }
I20260928 02:39:20.647609 17292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:20.647651 17292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:20.647698 17292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:20.647855 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 17214
I20260928 02:39:20.647960 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:20.648171 9736 external_mini_cluster.cc:1428] Setting key c1fdbcffc196f6e22fbcd2b3b89635a3
I20260928 02:39:20.648365 17292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9937968dbb6b4fda88022d40089c2d62" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38137 } }
I20260928 02:39:20.648473 17292 leader_election.cc:302] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9937968dbb6b4fda88022d40089c2d62; no voters:
I20260928 02:39:20.648646 17292 leader_election.cc:288] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:20.648687 17295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:20.648845 17295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [term 1 LEADER]: Becoming Leader. State: Replica: 9937968dbb6b4fda88022d40089c2d62, State: Running, Role: LEADER
I20260928 02:39:20.648932 17292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:20.648955 17295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9937968dbb6b4fda88022d40089c2d62" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38137 } }
I20260928 02:39:20.649350 17295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9937968dbb6b4fda88022d40089c2d62. Latest consensus state: current_term: 1 leader_uuid: "9937968dbb6b4fda88022d40089c2d62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9937968dbb6b4fda88022d40089c2d62" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38137 } } }
I20260928 02:39:20.649573 17295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:20.649499 17298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9937968dbb6b4fda88022d40089c2d62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9937968dbb6b4fda88022d40089c2d62" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38137 } } }
I20260928 02:39:20.649945 17298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:20.650081 17306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:20.650635 17306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:20.652208 17306 catalog_manager.cc:1347] Generated new cluster ID: ef82cb6a329d406996d2209af73b36c2
I20260928 02:39:20.652257 17306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:39:20 dist-test-slave-kkrh krb5kdc[17201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
I20260928 02:39:20.661000 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:20.648644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.9.130.62:38137)
0928 02:39:20.648803 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:20.648813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:20.648984 (+ 171us) server_negotiation.cc:407] Connection header received
0928 02:39:20.649203 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:20.649211 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:20.649265 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:20.649379 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:20.650471 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.651760 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.652875 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.653487 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.653901 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:20.656419 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:20.656423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:20.656423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:20.656435 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:20.658980 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:20.659379 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:20.659381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:20.659381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:20.659484 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:20.659719 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:20.659720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:20.659721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:20.660031 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:20.660569 (+ 538us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:20.660656 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:39:20.660788 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260928 02:39:20.667549 17306 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:20.668277 17306 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:20.672087 17306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62: Generated new TSK 0
I20260928 02:39:20.672277 17306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab.
Entry for principal oryx/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab.
I20260928 02:39:20.699384 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38137
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab
--principal=oryx/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:20.778272 17318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:20.778466 17318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:20.778489 17318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:20.779654 17318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:20.779692 17318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:20.779712 17318 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:39:20.779729 17318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:20.781195 17318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab
--principal=oryx/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.130.62:38137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:20.781427 17318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:20.781638 17318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:20.786479 17324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:20.786595 17323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:20.786613 17326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:20.786824 17318 server_base.cc:1034] running on GCE node
I20260928 02:39:20.787053 17318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:20.787259 17318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:20.788398 17318 hybrid_clock.cc:648] HybridClock initialized: now 1790563160788385 us; error 28 us; skew 500 ppm
Sep 28 02:39:20 dist-test-slave-kkrh krb5kdc[17201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563160, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:20.791019 17318 init.cc:377] Logged in from keytab as oryx/127.9.130.1@KRBTEST.COM (short username oryx)
I20260928 02:39:20.791457 17318 webserver.cc:460] Webserver started at http://127.9.130.1:36063/ using document root <none> and password file <none>
I20260928 02:39:20.791692 17318 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:20.791736 17318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:20.791929 17318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:20.792995 17318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "fc36762ef32440e48126c12cf1f3d255"
format_stamp: "Formatted at 2026-09-28 02:39:20 on dist-test-slave-kkrh"
server_key: "088f1bb015925d2c8a7985efe93b3dab"
server_key_iv: "16c168f7b133e3b9387ecd28957a957a"
server_key_version: "encryptionkey@0"
I20260928 02:39:20.793372 17318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "fc36762ef32440e48126c12cf1f3d255"
format_stamp: "Formatted at 2026-09-28 02:39:20 on dist-test-slave-kkrh"
server_key: "088f1bb015925d2c8a7985efe93b3dab"
server_key_iv: "16c168f7b133e3b9387ecd28957a957a"
server_key_version: "encryptionkey@0"
I20260928 02:39:20.794654 17318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:39:20.795392 17333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:20.795560 17318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:20.795655 17318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "fc36762ef32440e48126c12cf1f3d255"
format_stamp: "Formatted at 2026-09-28 02:39:20 on dist-test-slave-kkrh"
server_key: "088f1bb015925d2c8a7985efe93b3dab"
server_key_iv: "16c168f7b133e3b9387ecd28957a957a"
server_key_version: "encryptionkey@0"
I20260928 02:39:20.795735 17318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:20.803896 17318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:20.806082 17318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:20.806232 17318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:20.806735 17318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:20.806802 17318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:20.806841 17318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:20.806897 17318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:39:20 dist-test-slave-kkrh krb5kdc[17201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563160, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
I20260928 02:39:20.813917 17318 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:46669
I20260928 02:39:20.814059 17448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:46669 every 8 connection(s)
I20260928 02:39:20.814324 17318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:39:20.817364 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:20.807270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:40643 (local address 127.9.130.62:38137)
0928 02:39:20.807339 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:20.807341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:20.807521 (+ 180us) server_negotiation.cc:407] Connection header received
0928 02:39:20.808098 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:20.808100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:20.808114 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:20.808164 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:20.809012 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.809855 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.811527 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.811893 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.811963 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:20.814377 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:20.814379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:20.814379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:20.814386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:20.816023 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:20.816428 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:20.816429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:20.816429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:20.816488 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:20.816740 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:20.816741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:20.816741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:20.816853 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:20.816934 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:20.817195 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 02:39:20.817279 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:39:20.817535 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:20.807247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38137 (local address 127.9.130.1:40643)
0928 02:39:20.807414 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:20.807450 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:20.807994 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:20.808264 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:20.808268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:20.808614 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:20.808898 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:20.808904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.809985 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.809986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:20.811418 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:20.811421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.811979 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.811980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:20.812124 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:20.812712 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:20.812720 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:20.814266 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:20.816113 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:20.816119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:20.816122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:20.816334 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:20.816554 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:20.816555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:20.816555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:20.816664 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:20.816924 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:20.816929 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:20.817061 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 02:39:20.817161 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:39:20.817308 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260928 02:39:20.818070 17449 heartbeater.cc:347] Connected to a master server at 127.9.130.62:38137
I20260928 02:39:20.818154 17449 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:20.818323 17449 heartbeater.cc:511] Master 127.9.130.62:38137 requested a full tablet report, sending...
I20260928 02:39:20.818835 17245 ts_manager.cc:194] Registered new tserver with Master: fc36762ef32440e48126c12cf1f3d255 (127.9.130.1:46669)
I20260928 02:39:20.819535 17245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:40643
I20260928 02:39:20.823596 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 17318
I20260928 02:39:20.823678 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:39:20.823881 9736 external_mini_cluster.cc:1428] Setting key 22a5319a3fb87706a053afc5c3111781
I20260928 02:39:20.824887 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:20.819010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38137 (local address 127.9.130.1:44299)
0928 02:39:20.819120 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:20.819130 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:20.819185 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:20.819304 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:20.819305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:20.819468 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:20.819519 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:20.819520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.820242 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.820243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:20.820775 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:20.820776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.821150 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.821151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:20.821218 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:20.821597 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:20.821601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:20.821877 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:20.823209 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:20.823210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:20.823210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:20.823371 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:20.824151 (+ 780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:20.824152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:20.824152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:20.824219 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:20.824519 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:20.824519 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:20.824574 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 02:39:20.824661 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:39:20.824780 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":59,"threads_started":1}
I20260928 02:39:20.825021 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:20.819066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:44299 (local address 127.9.130.62:38137)
0928 02:39:20.819119 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:20.819120 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:20.819170 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:39:20.819235 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:20.819236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:20.819245 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:20.819299 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:20.819600 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.820155 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.820871 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.821069 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.821149 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:20.822008 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:20.822009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:20.822010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:20.822016 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:20.823115 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:20.823996 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:20.823997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:20.823998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:20.824074 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:20.824304 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:20.824305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:20.824305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:20.824451 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:20.824525 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:20.824797 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 02:39:20.824914 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
WARNING: no policy specified for oryx/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab.
Entry for principal oryx/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab.
I20260928 02:39:20.849076 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38137
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab
--principal=oryx/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:20.929975 17458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:20.930207 17458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:20.930243 17458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:20.931442 17458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:20.931502 17458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:20.931536 17458 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:39:20.931574 17458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:20.933099 17458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab
--principal=oryx/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.130.62:38137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:20.933355 17458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:20.933578 17458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:20.938566 17466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:20.938621 17463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:20.938719 17458 server_base.cc:1034] running on GCE node
W20260928 02:39:20.938788 17464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:20.939033 17458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:20.939306 17458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:20.940459 17458 hybrid_clock.cc:648] HybridClock initialized: now 1790563160940442 us; error 27 us; skew 500 ppm
Sep 28 02:39:20 dist-test-slave-kkrh krb5kdc[17201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563160, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:20.942893 17458 init.cc:377] Logged in from keytab as oryx/127.9.130.2@KRBTEST.COM (short username oryx)
I20260928 02:39:20.943336 17458 webserver.cc:460] Webserver started at http://127.9.130.2:39677/ using document root <none> and password file <none>
I20260928 02:39:20.943519 17458 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:20.943558 17458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:20.943645 17458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:20.944743 17458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2"
format_stamp: "Formatted at 2026-09-28 02:39:20 on dist-test-slave-kkrh"
server_key: "acf6ca3ef932c252ea83162a05f9918a"
server_key_iv: "30caba01b94a8b01faaf967df9653adc"
server_key_version: "encryptionkey@0"
I20260928 02:39:20.945052 17458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2"
format_stamp: "Formatted at 2026-09-28 02:39:20 on dist-test-slave-kkrh"
server_key: "acf6ca3ef932c252ea83162a05f9918a"
server_key_iv: "30caba01b94a8b01faaf967df9653adc"
server_key_version: "encryptionkey@0"
I20260928 02:39:20.946308 17458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:39:20.947003 17473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:20.947163 17458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:20.947252 17458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2"
format_stamp: "Formatted at 2026-09-28 02:39:20 on dist-test-slave-kkrh"
server_key: "acf6ca3ef932c252ea83162a05f9918a"
server_key_iv: "30caba01b94a8b01faaf967df9653adc"
server_key_version: "encryptionkey@0"
I20260928 02:39:20.947311 17458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:20.959945 17458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:20.962162 17458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:20.962272 17458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:20.962708 17458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:20.962749 17458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:20.962772 17458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:20.962808 17458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:39:20 dist-test-slave-kkrh krb5kdc[17201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563160, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
I20260928 02:39:20.969830 17458 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:43423
I20260928 02:39:20.969878 17588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:43423 every 8 connection(s)
I20260928 02:39:20.970175 17458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:39:20.973156 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:20.963198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:53497 (local address 127.9.130.62:38137)
0928 02:39:20.963269 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:20.963272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:20.963448 (+ 176us) server_negotiation.cc:407] Connection header received
0928 02:39:20.964029 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:20.964031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:20.964045 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:20.964135 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:20.964797 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.965613 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.967279 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.967610 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.967682 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:20.970313 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:20.970315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:20.970315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:20.970323 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:20.971739 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:20.972164 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:20.972165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:20.972166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:20.972227 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:20.972478 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:20.972479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:20.972479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:20.972597 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:20.972669 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:20.972979 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 02:39:20.973056 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:39:20.973323 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:20.963174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38137 (local address 127.9.130.2:53497)
0928 02:39:20.963319 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:20.963359 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:20.963927 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:20.964222 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:20.964227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:20.964508 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:20.964701 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:20.964706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.965725 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.965727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:20.967187 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:20.967190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.967724 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.967725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:20.967899 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:20.968389 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:20.968399 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:20.970233 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:20.971855 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:20.971862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:20.971865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:20.972088 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:20.972293 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:20.972294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:20.972294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:20.972406 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:20.972672 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:20.972679 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:20.972813 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 02:39:20.972942 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 02:39:20.973098 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260928 02:39:20.973590 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 17458
I20260928 02:39:20.973686 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:39:20.973809 17589 heartbeater.cc:347] Connected to a master server at 127.9.130.62:38137
I20260928 02:39:20.973889 9736 external_mini_cluster.cc:1428] Setting key 86dce014d318e878c0a93c002fd3bba0
I20260928 02:39:20.973891 17589 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:20.974099 17589 heartbeater.cc:511] Master 127.9.130.62:38137 requested a full tablet report, sending...
I20260928 02:39:20.974813 17246 ts_manager.cc:194] Registered new tserver with Master: 8e0dbbf30e024ddb978e257ee5e9d1d2 (127.9.130.2:43423)
I20260928 02:39:20.975340 17246 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:53497
I20260928 02:39:20.981031 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:20.974890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:34301 (local address 127.9.130.62:38137)
0928 02:39:20.974930 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:20.974931 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:20.975055 (+ 124us) server_negotiation.cc:407] Connection header received
0928 02:39:20.975192 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:20.975193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:20.975203 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:20.975246 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:20.975584 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.976295 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.977314 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.977516 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.977603 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:20.978534 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:20.978535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:20.978536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:20.978542 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:20.979888 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:20.980215 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:20.980215 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:20.980216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:20.980267 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:20.980467 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:20.980468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:20.980469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:20.980613 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:20.980713 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:20.980871 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:39:20.980943 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:39:20.981046 17595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:20.974848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38137 (local address 127.9.130.2:34301)
0928 02:39:20.974980 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:20.974989 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:20.975110 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:20.975259 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:20.975260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:20.975417 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:20.975503 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:20.975505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.976389 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.976390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:20.977223 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:20.977225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:20.977605 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:20.977606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:20.977667 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:20.978139 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:20.978145 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:20.978442 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:20.979960 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:20.979961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:20.979962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:20.980131 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:20.980339 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:20.980340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:20.980341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:20.980396 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:20.980707 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:20.980708 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:20.980762 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 02:39:20.980848 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:39:20.980945 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for oryx/127.9.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab.
Entry for principal oryx/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab.
I20260928 02:39:20.999334 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38137
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab
--principal=oryx/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:21.079602 17598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:21.079802 17598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:21.079841 17598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:21.080998 17598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:21.081040 17598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:21.081061 17598 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:39:21.081079 17598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:21.082535 17598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab
--principal=oryx/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.130.62:38137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:21.082772 17598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:21.082985 17598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:21.087802 17603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:21.087877 17604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:21.087900 17598 server_base.cc:1034] running on GCE node
W20260928 02:39:21.087816 17606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:21.088294 17598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:21.088490 17598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:21.089627 17598 hybrid_clock.cc:648] HybridClock initialized: now 1790563161089600 us; error 41 us; skew 500 ppm
Sep 28 02:39:21 dist-test-slave-kkrh krb5kdc[17201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563161, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:21.092422 17598 init.cc:377] Logged in from keytab as oryx/127.9.130.3@KRBTEST.COM (short username oryx)
I20260928 02:39:21.092844 17598 webserver.cc:460] Webserver started at http://127.9.130.3:42019/ using document root <none> and password file <none>
I20260928 02:39:21.093065 17598 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:21.093108 17598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:21.093263 17598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:21.094380 17598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "2c99680fca5a49f6a3b355ee0c0b853b"
format_stamp: "Formatted at 2026-09-28 02:39:21 on dist-test-slave-kkrh"
server_key: "11a52851439baca6dce63f0cff971f13"
server_key_iv: "5b8e86b9f47dac68b1e04f84a5927911"
server_key_version: "encryptionkey@0"
I20260928 02:39:21.094731 17598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "2c99680fca5a49f6a3b355ee0c0b853b"
format_stamp: "Formatted at 2026-09-28 02:39:21 on dist-test-slave-kkrh"
server_key: "11a52851439baca6dce63f0cff971f13"
server_key_iv: "5b8e86b9f47dac68b1e04f84a5927911"
server_key_version: "encryptionkey@0"
I20260928 02:39:21.096107 17598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:39:21.096801 17613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:21.096964 17598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:21.097066 17598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "2c99680fca5a49f6a3b355ee0c0b853b"
format_stamp: "Formatted at 2026-09-28 02:39:21 on dist-test-slave-kkrh"
server_key: "11a52851439baca6dce63f0cff971f13"
server_key_iv: "5b8e86b9f47dac68b1e04f84a5927911"
server_key_version: "encryptionkey@0"
I20260928 02:39:21.097146 17598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:21.123077 17598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:21.125516 17598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:21.125705 17598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:21.126392 17598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:21.126451 17598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:21.126490 17598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:21.126518 17598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:39:21 dist-test-slave-kkrh krb5kdc[17201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563161, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
I20260928 02:39:21.133225 17598 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:32961
I20260928 02:39:21.133296 17728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:32961 every 8 connection(s)
I20260928 02:39:21.133585 17598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:39:21.134112 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 17598
I20260928 02:39:21.134238 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:39:21.134456 9736 external_mini_cluster.cc:1428] Setting key 3b8f027b69b1868cf6cc1526d5bd3539
I20260928 02:39:21.136276 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.126697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:39303 (local address 127.9.130.62:38137)
0928 02:39:21.126743 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.126746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.126989 (+ 243us) server_negotiation.cc:407] Connection header received
0928 02:39:21.127464 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.127465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.127478 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.127566 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:21.128416 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.129276 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.130832 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.131164 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.131246 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.133438 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:21.133440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:21.133440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:21.133448 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:21.135093 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:21.135504 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:21.135505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:21.135505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:21.135558 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:21.135734 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:21.135735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:21.135735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:21.135878 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:21.135942 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.136145 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.136214 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:39:21.136404 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.126706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38137 (local address 127.9.130.3:39303)
0928 02:39:21.126845 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:21.126883 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:21.127363 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:21.127617 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:21.127620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:21.128019 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:21.128308 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.128314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.129370 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.129371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.130736 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.130739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.131257 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.131258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.131388 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.131895 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:21.131905 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:21.133333 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:21.135163 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:21.135169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:21.135170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:21.135443 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:21.135600 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:21.135601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:21.135601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:21.135683 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:21.135933 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:21.135936 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:21.136040 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 02:39:21.136118 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:39:21.136228 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260928 02:39:21.136845 17729 heartbeater.cc:347] Connected to a master server at 127.9.130.62:38137
I20260928 02:39:21.136919 17729 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:21.137104 17729 heartbeater.cc:511] Master 127.9.130.62:38137 requested a full tablet report, sending...
I20260928 02:39:21.137450 17245 ts_manager.cc:194] Registered new tserver with Master: 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961)
I20260928 02:39:21.137925 17245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:39303
I20260928 02:39:21.142567 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.137871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:38499 (local address 127.9.130.62:38137)
0928 02:39:21.137906 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.137908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.138022 (+ 114us) server_negotiation.cc:407] Connection header received
0928 02:39:21.138069 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.138070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.138079 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.138116 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:21.138501 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.139010 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.139654 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.139859 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.139918 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.140642 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:21.140643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:21.140644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:21.140649 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:21.141812 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:21.142092 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:21.142093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:21.142093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:21.142137 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:21.142256 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:21.142256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:21.142257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:21.142331 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:21.142373 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.142454 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.142512 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:39:21.142575 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.137824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38137 (local address 127.9.130.3:38499)
0928 02:39:21.137951 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:21.137961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:21.138015 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:21.138234 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:21.138235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:21.138377 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:21.138428 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.138429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.139094 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.139095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.139582 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.139584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.139912 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.139912 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.139968 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.140311 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:21.140315 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:21.140558 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:21.141880 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:21.141881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:21.141881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:21.142034 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:21.142177 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:21.142177 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:21.142178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:21.142221 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:21.142365 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:21.142365 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:21.142396 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:39:21.142441 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:39:21.142508 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260928 02:39:21.146010 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:39:21 dist-test-slave-kkrh krb5kdc[17201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563161, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:39:21 dist-test-slave-kkrh krb5kdc[17201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:21 dist-test-slave-kkrh krb5kdc[17201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:39:21.166934 17297 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:21.161653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.9.130.62:38137)
0928 02:39:21.161715 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.161717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.161725 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:21.161758 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.161759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.161767 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.161834 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:21.162396 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.162885 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.163596 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.163788 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.163894 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.166843 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60176: BlockingRecv error: recv got EOF from 127.0.0.1:60176 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 02:39:21 dist-test-slave-kkrh krb5kdc[17201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563161, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
I20260928 02:39:21.174201 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.167605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.9.130.62:38137)
0928 02:39:21.167665 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.167667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.167724 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:39:21.167798 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.167798 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.167808 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.167886 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:21.168350 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.168806 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.169485 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.169671 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.169746 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.171666 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:21.171667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:21.171668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:21.171674 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:21.173043 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:21.173371 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:21.173372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:21.173372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:21.173433 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:21.173639 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:21.173640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:21.173640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:21.173751 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:21.173854 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.173977 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.174046 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:39:21.176004 17245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:39:21.177465 17245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:39:21.183717 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.179708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:43423 (local address 127.0.0.1:59000)
0928 02:39:21.180149 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:21.180158 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:21.180227 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:21.180454 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:21.180455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:21.180457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:21.180508 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.180510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.181331 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.181332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.182540 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.182543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.183372 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.183373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.183448 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.183462 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 02:39:21.183564 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:39:21.183618 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260928 02:39:21.183723 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.179944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.9.130.2:43423)
0928 02:39:21.180131 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.180136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.180218 (+ 82us) server_negotiation.cc:407] Connection header received
0928 02:39:21.180303 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.180305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.180320 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.180391 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:21.180587 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.181235 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.182636 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.183289 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.183373 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.183453 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.183582 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.183621 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":119,"threads_started":1}
I20260928 02:39:21.183735 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.179524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46669 (local address 127.0.0.1:47120)
0928 02:39:21.179782 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:21.179849 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:21.179925 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:21.180143 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:21.180145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:21.180147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:21.180206 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.180207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.181609 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.181610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.182518 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.182520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.183469 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.183470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.183534 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.183538 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:39:21.183585 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:39:21.183638 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260928 02:39:21.184088 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.179515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:32961 (local address 127.0.0.1:37174)
0928 02:39:21.179781 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:21.179852 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:21.179924 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:21.180471 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:21.180472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:21.180474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:21.180512 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.180514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.181702 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.181703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.182725 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.182726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.183857 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.183858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.183922 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.183926 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:39:21.183989 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:39:21.184024 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":55,"threads_started":1}
I20260928 02:39:21.184098 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.179626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47120 (local address 127.9.130.1:46669)
0928 02:39:21.179859 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.179863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.179902 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:39:21.180015 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.180017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.180031 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.180136 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:21.180357 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.181506 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.182605 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.183292 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.183365 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.183434 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.183958 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.184010 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":63,"threads_started":1}
I20260928 02:39:21.184418 17523 tablet_service.cc:1467] Processing CreateTablet for tablet 31fed78d3c0040dfbeab6f509559c55c (DEFAULT_TABLE table=test-table [id=6da3e50d173c4323b3e6dd7ec76ccec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:21.184744 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.179604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.9.130.3:32961)
0928 02:39:21.179766 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.179771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.179923 (+ 152us) server_negotiation.cc:407] Connection header received
0928 02:39:21.180230 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.180234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.180250 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.180413 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:21.180586 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.181622 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.182800 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.183488 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.183578 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.183644 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.184590 (+ 946us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.184650 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260928 02:39:21.184777 17523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31fed78d3c0040dfbeab6f509559c55c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.184847 17383 tablet_service.cc:1467] Processing CreateTablet for tablet 31fed78d3c0040dfbeab6f509559c55c (DEFAULT_TABLE table=test-table [id=6da3e50d173c4323b3e6dd7ec76ccec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:21.185194 17383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31fed78d3c0040dfbeab6f509559c55c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.187795 17757 tablet_bootstrap.cc:492] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2: Bootstrap starting.
I20260928 02:39:21.188217 17758 tablet_bootstrap.cc:492] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255: Bootstrap starting.
I20260928 02:39:21.188577 17757 tablet_bootstrap.cc:654] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.188585 17663 tablet_service.cc:1467] Processing CreateTablet for tablet 31fed78d3c0040dfbeab6f509559c55c (DEFAULT_TABLE table=test-table [id=6da3e50d173c4323b3e6dd7ec76ccec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:21.188851 17758 tablet_bootstrap.cc:654] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.188916 17757 log.cc:824] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:21.188975 17663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31fed78d3c0040dfbeab6f509559c55c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.189173 17758 log.cc:824] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:21.189666 17757 tablet_bootstrap.cc:492] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2: No bootstrap required, opened a new log
I20260928 02:39:21.189718 17757 ts_tablet_manager.cc:1397] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:39:21.191221 17757 raft_consensus.cc:348] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } }
I20260928 02:39:21.191392 17757 raft_consensus.cc:374] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.191416 17757 raft_consensus.cc:729] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0dbbf30e024ddb978e257ee5e9d1d2, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.191547 17757 consensus_queue.cc:260] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } }
I20260928 02:39:21.191891 17589 heartbeater.cc:503] Master 127.9.130.62:38137 was elected leader, sending a full tablet report...
I20260928 02:39:21.191929 17757 ts_tablet_manager.cc:1428] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:39:21.192452 17758 tablet_bootstrap.cc:492] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255: No bootstrap required, opened a new log
I20260928 02:39:21.192517 17762 tablet_bootstrap.cc:492] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b: Bootstrap starting.
I20260928 02:39:21.192574 17758 ts_tablet_manager.cc:1397] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:39:21.193151 17762 tablet_bootstrap.cc:654] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.193416 17762 log.cc:824] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:21.194062 17762 tablet_bootstrap.cc:492] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b: No bootstrap required, opened a new log
I20260928 02:39:21.194139 17762 ts_tablet_manager.cc:1397] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:39:21.194322 17758 raft_consensus.cc:348] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } }
I20260928 02:39:21.194490 17758 raft_consensus.cc:374] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.194531 17758 raft_consensus.cc:729] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc36762ef32440e48126c12cf1f3d255, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.194653 17758 consensus_queue.cc:260] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } }
I20260928 02:39:21.194903 17758 ts_tablet_manager.cc:1428] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:39:21.194970 17449 heartbeater.cc:503] Master 127.9.130.62:38137 was elected leader, sending a full tablet report...
I20260928 02:39:21.196027 17762 raft_consensus.cc:348] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } }
I20260928 02:39:21.196138 17762 raft_consensus.cc:374] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.196157 17762 raft_consensus.cc:729] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c99680fca5a49f6a3b355ee0c0b853b, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.196223 17762 consensus_queue.cc:260] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } }
I20260928 02:39:21.196395 17762 ts_tablet_manager.cc:1428] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 02:39:21.196450 17729 heartbeater.cc:503] Master 127.9.130.62:38137 was elected leader, sending a full tablet report...
W20260928 02:39:21.220733 17590 tablet.cc:2369] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:39:21.237896 17761 raft_consensus.cc:482] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:39:21.238058 17761 raft_consensus.cc:504] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } }
I20260928 02:39:21.238451 17761 leader_election.cc:288] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961), fc36762ef32440e48126c12cf1f3d255 (127.9.130.1:46669)
I20260928 02:39:21.241600 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.238778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:60327 (local address 127.9.130.1:46669)
0928 02:39:21.238817 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.238819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.239016 (+ 197us) server_negotiation.cc:407] Connection header received
0928 02:39:21.239045 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.239046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.239057 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.239101 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:21.239259 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.239726 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.240914 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.241259 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.241338 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.241348 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.241493 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.241525 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:39:21.241631 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.238726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:45227 (local address 127.9.130.3:32961)
0928 02:39:21.238772 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.238774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.238785 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:39:21.238861 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.238862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.238875 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.238945 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:21.239292 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.239766 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.240920 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.241282 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.241363 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.241372 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.241527 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.241563 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:39:21.241639 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.238641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:32961 (local address 127.9.130.2:45227)
0928 02:39:21.238703 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:21.238717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:21.238775 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:21.239144 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:21.239145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:21.239147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:21.239203 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.239204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.239885 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.239886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.240839 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.240841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.241362 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.241363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.241445 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.241448 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:21.241510 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:39:21.241555 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":19}
I20260928 02:39:21.241645 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.238678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46669 (local address 127.9.130.2:60327)
0928 02:39:21.238865 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:21.238875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:21.238934 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:21.239117 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:21.239118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:21.239119 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:21.239182 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.239184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.239805 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.239806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.240828 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.240831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.241348 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.241349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.241419 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.241422 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:21.241486 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:39:21.241522 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":124,"threads_started":1}
I20260928 02:39:21.242065 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fed78d3c0040dfbeab6f509559c55c" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" is_pre_election: true
I20260928 02:39:21.242241 17683 raft_consensus.cc:2454] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 0.
I20260928 02:39:21.242205 17403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fed78d3c0040dfbeab6f509559c55c" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc36762ef32440e48126c12cf1f3d255" is_pre_election: true
I20260928 02:39:21.242326 17403 raft_consensus.cc:2454] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 0.
I20260928 02:39:21.242447 17474 leader_election.cc:302] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c99680fca5a49f6a3b355ee0c0b853b, 8e0dbbf30e024ddb978e257ee5e9d1d2; no voters:
I20260928 02:39:21.242571 17761 raft_consensus.cc:2781] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:39:21.242619 17761 raft_consensus.cc:482] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:39:21.242633 17761 raft_consensus.cc:3037] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.243510 17761 raft_consensus.cc:504] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } }
I20260928 02:39:21.243649 17761 leader_election.cc:288] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 election: Requested vote from peers 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961), fc36762ef32440e48126c12cf1f3d255 (127.9.130.1:46669)
I20260928 02:39:21.243798 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fed78d3c0040dfbeab6f509559c55c" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c99680fca5a49f6a3b355ee0c0b853b"
I20260928 02:39:21.243800 17403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fed78d3c0040dfbeab6f509559c55c" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc36762ef32440e48126c12cf1f3d255"
I20260928 02:39:21.243892 17683 raft_consensus.cc:3037] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.243914 17403 raft_consensus.cc:3037] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.244563 17683 raft_consensus.cc:2454] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 1.
I20260928 02:39:21.244688 17403 raft_consensus.cc:2454] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 1.
I20260928 02:39:21.244720 17474 leader_election.cc:302] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c99680fca5a49f6a3b355ee0c0b853b, 8e0dbbf30e024ddb978e257ee5e9d1d2; no voters:
I20260928 02:39:21.244810 17761 raft_consensus.cc:2781] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:21.244980 17761 raft_consensus.cc:686] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 LEADER]: Becoming Leader. State: Replica: 8e0dbbf30e024ddb978e257ee5e9d1d2, State: Running, Role: LEADER
I20260928 02:39:21.245087 17761 consensus_queue.cc:237] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } }
I20260928 02:39:21.245857 17245 catalog_manager.cc:5495] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0dbbf30e024ddb978e257ee5e9d1d2 (127.9.130.2). New cstate: current_term: 1 leader_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } health_report { overall_health: HEALTHY } } }
I20260928 02:39:21.260092 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.257849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.9.130.2:43423)
0928 02:39:21.257896 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.257898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.257905 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:39:21.257977 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.257978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.257994 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.258056 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:39:21.258185 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.258629 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.259352 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.259519 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.259567 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.259747 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:39:21.259881 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:39:21.259958 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.259990 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.260005 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:39:21.261711 17403 raft_consensus.cc:1261] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Refusing update from remote peer 8e0dbbf30e024ddb978e257ee5e9d1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:21.261718 17683 raft_consensus.cc:1261] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Refusing update from remote peer 8e0dbbf30e024ddb978e257ee5e9d1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:21.262037 17767 consensus_queue.cc:1035] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:21.262166 17761 consensus_queue.cc:1035] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:21.265594 17770 mvcc.cc:205] Tried to move back new op lower bound from 7334146708525404160 to 7334146708460343296. Current Snapshot: MvccSnapshot[applied={T|T < 7334146708525404160 or (T in {7334146708525404160})}]
I20260928 02:39:21.266731 17246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:21.266819 17246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:21.267602 17246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62: Sending DeleteTablet for 3 replicas of tablet 31fed78d3c0040dfbeab6f509559c55c
I20260928 02:39:21.268013 17383 tablet_service.cc:1514] Processing DeleteTablet for tablet 31fed78d3c0040dfbeab6f509559c55c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:21 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:47120
I20260928 02:39:21.268019 17523 tablet_service.cc:1514] Processing DeleteTablet for tablet 31fed78d3c0040dfbeab6f509559c55c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:21 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:59000
I20260928 02:39:21.268239 17663 tablet_service.cc:1514] Processing DeleteTablet for tablet 31fed78d3c0040dfbeab6f509559c55c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:21 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:37174
I20260928 02:39:21.268307 17780 tablet_replica.cc:330] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2: stopping tablet replica
I20260928 02:39:21.268433 17781 tablet_replica.cc:330] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b: stopping tablet replica
I20260928 02:39:21.268512 17780 raft_consensus.cc:2229] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:39:21.268518 17781 raft_consensus.cc:2229] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:21.268472 17246 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:39:21.268611 17780 raft_consensus.cc:2258] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 02:39:21.268612 17246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:39:21.268621 17781 raft_consensus.cc:2258] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:21.268764 17779 tablet_replica.cc:330] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255: stopping tablet replica
I20260928 02:39:21.268846 17779 raft_consensus.cc:2229] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:21.268942 17779 raft_consensus.cc:2258] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:21.269590 17779 ts_tablet_manager.cc:1905] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:21.269806 17780 ts_tablet_manager.cc:1905] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:21.269842 17383 tablet_service.cc:1467] Processing CreateTablet for tablet 0aeb8457f94c4e84ace17aec180cabf2 (DEFAULT_TABLE table=test-table [id=bffc3f91c0984a35a06bc02d912a598e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:21.269938 17383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aeb8457f94c4e84ace17aec180cabf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.271214 17758 tablet_bootstrap.cc:492] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255: Bootstrap starting.
I20260928 02:39:21.271482 17779 ts_tablet_manager.cc:1918] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:21.271486 17780 ts_tablet_manager.cc:1918] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:21.271541 17779 log.cc:1196] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/wal/wals/31fed78d3c0040dfbeab6f509559c55c
I20260928 02:39:21.271548 17780 log.cc:1196] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/31fed78d3c0040dfbeab6f509559c55c
I20260928 02:39:21.271602 17758 tablet_bootstrap.cc:654] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.271798 17779 ts_tablet_manager.cc:1939] T 31fed78d3c0040dfbeab6f509559c55c P fc36762ef32440e48126c12cf1f3d255: Deleting consensus metadata
I20260928 02:39:21.271844 17780 ts_tablet_manager.cc:1939] T 31fed78d3c0040dfbeab6f509559c55c P 8e0dbbf30e024ddb978e257ee5e9d1d2: Deleting consensus metadata
I20260928 02:39:21.272143 17523 tablet_service.cc:1467] Processing CreateTablet for tablet 0aeb8457f94c4e84ace17aec180cabf2 (DEFAULT_TABLE table=test-table [id=bffc3f91c0984a35a06bc02d912a598e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:21.272238 17523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aeb8457f94c4e84ace17aec180cabf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.272224 17663 tablet_service.cc:1467] Processing CreateTablet for tablet 0aeb8457f94c4e84ace17aec180cabf2 (DEFAULT_TABLE table=test-table [id=bffc3f91c0984a35a06bc02d912a598e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:21.272346 17663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aeb8457f94c4e84ace17aec180cabf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.273092 17757 tablet_bootstrap.cc:492] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Bootstrap starting.
I20260928 02:39:21.273520 17757 tablet_bootstrap.cc:654] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.273550 17230 catalog_manager.cc:4841] TS 8e0dbbf30e024ddb978e257ee5e9d1d2 (127.9.130.2:43423): tablet 31fed78d3c0040dfbeab6f509559c55c (table test-table [id=6da3e50d173c4323b3e6dd7ec76ccec6]) successfully deleted
I20260928 02:39:21.273777 17762 tablet_bootstrap.cc:492] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b: Bootstrap starting.
I20260928 02:39:21.273957 17758 tablet_bootstrap.cc:492] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255: No bootstrap required, opened a new log
I20260928 02:39:21.274019 17758 ts_tablet_manager.cc:1397] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:39:21.274161 17762 tablet_bootstrap.cc:654] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.274184 17758 raft_consensus.cc:348] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.274271 17758 raft_consensus.cc:374] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.274299 17758 raft_consensus.cc:729] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc36762ef32440e48126c12cf1f3d255, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.274344 17758 consensus_queue.cc:260] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.274441 17758 ts_tablet_manager.cc:1428] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:21.272141 17781 ts_tablet_manager.cc:1905] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:21.275101 17757 tablet_bootstrap.cc:492] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2: No bootstrap required, opened a new log
I20260928 02:39:21.275158 17757 ts_tablet_manager.cc:1397] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:39:21.275296 17757 raft_consensus.cc:348] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.275368 17757 raft_consensus.cc:374] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.275395 17757 raft_consensus.cc:729] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0dbbf30e024ddb978e257ee5e9d1d2, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.275434 17757 consensus_queue.cc:260] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.275516 17757 ts_tablet_manager.cc:1428] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:21.275907 17781 ts_tablet_manager.cc:1918] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:21.275974 17781 log.cc:1196] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/31fed78d3c0040dfbeab6f509559c55c
I20260928 02:39:21.276088 17232 catalog_manager.cc:4841] TS fc36762ef32440e48126c12cf1f3d255 (127.9.130.1:46669): tablet 31fed78d3c0040dfbeab6f509559c55c (table test-table [id=6da3e50d173c4323b3e6dd7ec76ccec6]) successfully deleted
I20260928 02:39:21.276252 17781 ts_tablet_manager.cc:1939] T 31fed78d3c0040dfbeab6f509559c55c P 2c99680fca5a49f6a3b355ee0c0b853b: Deleting consensus metadata
I20260928 02:39:21.276454 17762 tablet_bootstrap.cc:492] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b: No bootstrap required, opened a new log
I20260928 02:39:21.276506 17762 ts_tablet_manager.cc:1397] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 02:39:21.276640 17762 raft_consensus.cc:348] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.276707 17762 raft_consensus.cc:374] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.276728 17762 raft_consensus.cc:729] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c99680fca5a49f6a3b355ee0c0b853b, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.276767 17762 consensus_queue.cc:260] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.276852 17762 ts_tablet_manager.cc:1428] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:21.276983 17230 catalog_manager.cc:4841] TS 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961): tablet 31fed78d3c0040dfbeab6f509559c55c (table test-table [id=6da3e50d173c4323b3e6dd7ec76ccec6]) successfully deleted
I20260928 02:39:21.284852 17767 raft_consensus.cc:482] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:39:21.284924 17767 raft_consensus.cc:504] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.285063 17767 leader_election.cc:288] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc36762ef32440e48126c12cf1f3d255 (127.9.130.1:46669), 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961)
I20260928 02:39:21.285233 17403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aeb8457f94c4e84ace17aec180cabf2" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc36762ef32440e48126c12cf1f3d255" is_pre_election: true
I20260928 02:39:21.285312 17403 raft_consensus.cc:2454] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 0.
I20260928 02:39:21.285394 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aeb8457f94c4e84ace17aec180cabf2" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" is_pre_election: true
I20260928 02:39:21.285461 17683 raft_consensus.cc:2454] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 0.
I20260928 02:39:21.285569 17474 leader_election.cc:302] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c99680fca5a49f6a3b355ee0c0b853b, 8e0dbbf30e024ddb978e257ee5e9d1d2; no voters:
I20260928 02:39:21.285656 17767 raft_consensus.cc:2781] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:39:21.285701 17767 raft_consensus.cc:482] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:39:21.285737 17767 raft_consensus.cc:3037] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.286482 17767 raft_consensus.cc:504] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.286610 17767 leader_election.cc:288] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 election: Requested vote from peers fc36762ef32440e48126c12cf1f3d255 (127.9.130.1:46669), 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961)
I20260928 02:39:21.286736 17403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aeb8457f94c4e84ace17aec180cabf2" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc36762ef32440e48126c12cf1f3d255"
I20260928 02:39:21.286830 17403 raft_consensus.cc:3037] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.286864 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aeb8457f94c4e84ace17aec180cabf2" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c99680fca5a49f6a3b355ee0c0b853b"
I20260928 02:39:21.286923 17683 raft_consensus.cc:3037] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.287336 17403 raft_consensus.cc:2454] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 1.
I20260928 02:39:21.287519 17476 leader_election.cc:302] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0dbbf30e024ddb978e257ee5e9d1d2, fc36762ef32440e48126c12cf1f3d255; no voters:
I20260928 02:39:21.287552 17683 raft_consensus.cc:2454] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 1.
I20260928 02:39:21.287607 17767 raft_consensus.cc:2781] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:21.287654 17767 raft_consensus.cc:686] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 LEADER]: Becoming Leader. State: Replica: 8e0dbbf30e024ddb978e257ee5e9d1d2, State: Running, Role: LEADER
I20260928 02:39:21.287700 17767 consensus_queue.cc:237] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.288223 17245 catalog_manager.cc:5495] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0dbbf30e024ddb978e257ee5e9d1d2 (127.9.130.2). New cstate: current_term: 1 leader_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } health_report { overall_health: UNKNOWN } } }
W20260928 02:39:21.315310 17450 tablet.cc:2369] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:39:21 dist-test-slave-kkrh krb5kdc[17201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563160, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.62@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
I20260928 02:39:21.335687 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.328604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.9.130.62:38137)
0928 02:39:21.328668 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.328670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.328784 (+ 114us) server_negotiation.cc:407] Connection header received
0928 02:39:21.328876 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.328876 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.328893 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.328950 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:21.329414 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.329897 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.330596 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.330777 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.330875 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.333039 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:21.333040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:21.333041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:21.333057 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:21.334489 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:21.334895 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:21.334896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:21.334896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:21.334947 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:21.335156 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:21.335157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:21.335157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:21.335264 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:21.335336 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.335484 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.335573 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:39:21.335783 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.328539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38137 (local address 127.0.0.1:60200)
0928 02:39:21.328715 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:21.328724 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:21.328804 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:21.328954 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:21.328955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:21.329284 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:21.329331 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.329333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.329977 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.329978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.330500 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.330501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.330872 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.330872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.330921 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.331490 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:21.331495 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:21.332935 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:21.334609 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:21.334616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:21.334618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:21.334809 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:21.335025 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:21.335025 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:21.335026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:21.335087 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:21.335327 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:21.335333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:21.335375 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:39:21.335461 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:39:21.335641 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":87,"thread_start_us":80,"threads_started":1}
I20260928 02:39:21.337589 17245 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:60200:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:39:21.337821 17245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:39:21.339313 17383 tablet_service.cc:1467] Processing CreateTablet for tablet 5f213afa494f45a79e034dd952c06101 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4881ee1b690f45b89ee1208106541949]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:39:21.339385 17523 tablet_service.cc:1467] Processing CreateTablet for tablet 5f213afa494f45a79e034dd952c06101 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4881ee1b690f45b89ee1208106541949]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:39:21.339474 17383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f213afa494f45a79e034dd952c06101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.339567 17523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f213afa494f45a79e034dd952c06101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.339964 17663 tablet_service.cc:1467] Processing CreateTablet for tablet 5f213afa494f45a79e034dd952c06101 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4881ee1b690f45b89ee1208106541949]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:39:21.340142 17663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f213afa494f45a79e034dd952c06101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.341027 17757 tablet_bootstrap.cc:492] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Bootstrap starting.
I20260928 02:39:21.341075 17758 tablet_bootstrap.cc:492] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255: Bootstrap starting.
I20260928 02:39:21.341408 17757 tablet_bootstrap.cc:654] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.341460 17758 tablet_bootstrap.cc:654] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.341785 17762 tablet_bootstrap.cc:492] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b: Bootstrap starting.
I20260928 02:39:21.342193 17762 tablet_bootstrap.cc:654] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.342360 17757 tablet_bootstrap.cc:492] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2: No bootstrap required, opened a new log
I20260928 02:39:21.342391 17757 ts_tablet_manager.cc:1397] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:21.342528 17757 raft_consensus.cc:348] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.342612 17757 raft_consensus.cc:374] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.342638 17757 raft_consensus.cc:729] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0dbbf30e024ddb978e257ee5e9d1d2, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.342689 17757 consensus_queue.cc:260] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.342783 17757 ts_tablet_manager.cc:1428] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:21.342819 17762 tablet_bootstrap.cc:492] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b: No bootstrap required, opened a new log
I20260928 02:39:21.342859 17762 ts_tablet_manager.cc:1397] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:21.342916 17758 tablet_bootstrap.cc:492] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255: No bootstrap required, opened a new log
I20260928 02:39:21.342882 17767 tablet_replica.cc:439] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } }
I20260928 02:39:21.342952 17767 tablet_replica.cc:442] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:39:21.342959 17758 ts_tablet_manager.cc:1397] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:39:21.342999 17762 raft_consensus.cc:348] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.343055 17762 raft_consensus.cc:374] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.343072 17762 raft_consensus.cc:729] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c99680fca5a49f6a3b355ee0c0b853b, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.343115 17762 consensus_queue.cc:260] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.343110 17758 raft_consensus.cc:348] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.343173 17758 raft_consensus.cc:374] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.343194 17758 raft_consensus.cc:729] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc36762ef32440e48126c12cf1f3d255, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.343219 17762 ts_tablet_manager.cc:1428] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:21.343235 17758 consensus_queue.cc:260] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.343330 17758 ts_tablet_manager.cc:1428] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:21.343353 17765 tablet_replica.cc:439] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } }
I20260928 02:39:21.343431 17765 tablet_replica.cc:442] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:39:21.343463 17764 tablet_replica.cc:439] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } }
I20260928 02:39:21.343529 17764 tablet_replica.cc:442] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 02:39:21.384199 17730 tablet.cc:2369] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:39:21.453828 17764 raft_consensus.cc:482] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:39:21.453925 17764 raft_consensus.cc:504] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.454308 17764 leader_election.cc:288] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e0dbbf30e024ddb978e257ee5e9d1d2 (127.9.130.2:43423), 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961)
I20260928 02:39:21.457970 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.454574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:58345 (local address 127.9.130.2:43423)
0928 02:39:21.454672 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.454675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.454852 (+ 177us) server_negotiation.cc:407] Connection header received
0928 02:39:21.454900 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.454901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.454913 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.454986 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:21.455201 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.455979 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.457122 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.457678 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.457729 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.457738 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.457889 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.457919 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:39:21.458002 17795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.454537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:43423 (local address 127.9.130.1:58345)
0928 02:39:21.454736 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:21.454755 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:21.454844 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:21.455011 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:21.455012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:21.455015 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:21.455101 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.455104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.456057 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.456058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.457033 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.457036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.457748 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.457749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.457818 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.457822 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:39:21.457878 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:39:21.457916 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
I20260928 02:39:21.458261 17543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f213afa494f45a79e034dd952c06101" candidate_uuid: "fc36762ef32440e48126c12cf1f3d255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" is_pre_election: true
I20260928 02:39:21.458408 17543 raft_consensus.cc:2454] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc36762ef32440e48126c12cf1f3d255 in term 0.
I20260928 02:39:21.458429 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.455289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:32961 (local address 127.9.130.1:58337)
0928 02:39:21.455459 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:21.455468 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:21.455526 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:21.455675 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:21.455675 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:21.455677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:21.455740 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.455742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.456613 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.456613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.457500 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.457503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.458203 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.458204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.458262 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.458266 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:39:21.458323 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:39:21.458357 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":46,"threads_started":1}
I20260928 02:39:21.458455 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.455474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:58337 (local address 127.9.130.3:32961)
0928 02:39:21.455541 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.455543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.455551 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:21.455580 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.455580 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.455589 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.455669 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:21.455828 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.456526 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.457599 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.458108 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.458171 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.458180 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.458341 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.458383 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:39:21.458627 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f213afa494f45a79e034dd952c06101" candidate_uuid: "fc36762ef32440e48126c12cf1f3d255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" is_pre_election: true
I20260928 02:39:21.458652 17334 leader_election.cc:302] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0dbbf30e024ddb978e257ee5e9d1d2, fc36762ef32440e48126c12cf1f3d255; no voters:
I20260928 02:39:21.458720 17683 raft_consensus.cc:2454] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc36762ef32440e48126c12cf1f3d255 in term 0.
I20260928 02:39:21.458782 17764 raft_consensus.cc:2781] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:39:21.458827 17764 raft_consensus.cc:482] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:39:21.458853 17764 raft_consensus.cc:3037] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.459535 17764 raft_consensus.cc:504] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.459625 17764 leader_election.cc:288] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [CANDIDATE]: Term 1 election: Requested vote from peers 8e0dbbf30e024ddb978e257ee5e9d1d2 (127.9.130.2:43423), 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961)
I20260928 02:39:21.459852 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f213afa494f45a79e034dd952c06101" candidate_uuid: "fc36762ef32440e48126c12cf1f3d255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c99680fca5a49f6a3b355ee0c0b853b"
I20260928 02:39:21.459844 17543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f213afa494f45a79e034dd952c06101" candidate_uuid: "fc36762ef32440e48126c12cf1f3d255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2"
I20260928 02:39:21.459916 17543 raft_consensus.cc:3037] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.459916 17683 raft_consensus.cc:3037] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.460579 17683 raft_consensus.cc:2454] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc36762ef32440e48126c12cf1f3d255 in term 1.
I20260928 02:39:21.460703 17543 raft_consensus.cc:2454] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc36762ef32440e48126c12cf1f3d255 in term 1.
I20260928 02:39:21.460773 17334 leader_election.cc:302] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c99680fca5a49f6a3b355ee0c0b853b, fc36762ef32440e48126c12cf1f3d255; no voters:
I20260928 02:39:21.460880 17764 raft_consensus.cc:2781] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:21.461009 17764 raft_consensus.cc:686] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [term 1 LEADER]: Becoming Leader. State: Replica: fc36762ef32440e48126c12cf1f3d255, State: Running, Role: LEADER
I20260928 02:39:21.461088 17764 consensus_queue.cc:237] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.461414 17797 tablet_replica.cc:439] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255: TxnStatusTablet state changed. Reason: New leader fc36762ef32440e48126c12cf1f3d255. Latest consensus state: current_term: 1 leader_uuid: "fc36762ef32440e48126c12cf1f3d255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } }
I20260928 02:39:21.461517 17797 tablet_replica.cc:442] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:39:21.461683 17799 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:39:21.461790 17799 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:39:21.461794 17246 catalog_manager.cc:5495] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc36762ef32440e48126c12cf1f3d255 (127.9.130.1). New cstate: current_term: 1 leader_uuid: "fc36762ef32440e48126c12cf1f3d255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } health_report { overall_health: UNKNOWN } } }
I20260928 02:39:21.492925 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.489947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46669 (local address 127.0.0.1:47132)
0928 02:39:21.490029 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:21.490045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:21.490099 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:21.490286 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:21.490287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:21.490288 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:39:21.490344 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.490346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.491232 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.491234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.491890 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.491898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.492368 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.492369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.492426 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.492441 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:39:21.492771 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:39:21.492778 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:39:21.492823 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:39:21.492867 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:39:21.492931 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.490064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.9.130.1:46669)
0928 02:39:21.490129 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.490131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.490154 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:39:21.490188 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.490189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.490209 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.490286 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:39:21.490384 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.491124 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.491980 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.492293 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.492364 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.492545 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:39:21.492688 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:39:21.492778 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.492828 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.492852 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:39:21.494240 17683 raft_consensus.cc:1261] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Refusing update from remote peer fc36762ef32440e48126c12cf1f3d255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:21.494378 17543 raft_consensus.cc:1261] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Refusing update from remote peer fc36762ef32440e48126c12cf1f3d255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:21.494539 17797 consensus_queue.cc:1035] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:21.494758 17764 consensus_queue.cc:1035] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:21.495371 17765 tablet_replica.cc:439] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b: TxnStatusTablet state changed. Reason: New leader fc36762ef32440e48126c12cf1f3d255. Latest consensus state: current_term: 1 leader_uuid: "fc36762ef32440e48126c12cf1f3d255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } }
I20260928 02:39:21.495493 17765 tablet_replica.cc:442] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:39:21.495428 17767 tablet_replica.cc:439] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2: TxnStatusTablet state changed. Reason: New leader fc36762ef32440e48126c12cf1f3d255. Latest consensus state: current_term: 1 leader_uuid: "fc36762ef32440e48126c12cf1f3d255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } }
I20260928 02:39:21.495533 17767 tablet_replica.cc:442] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:39:21.496404 17764 tablet_replica.cc:439] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc36762ef32440e48126c12cf1f3d255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } }
I20260928 02:39:21.496528 17764 tablet_replica.cc:442] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:39:21.496624 17772 mvcc.cc:205] Tried to move back new op lower bound from 7334146709478207488 to 7334146709345091584. Current Snapshot: MvccSnapshot[applied={T|T < 7334146709478207488 or (T in {7334146709478207488})}]
I20260928 02:39:21.496661 17771 mvcc.cc:205] Tried to move back new op lower bound from 7334146709478207488 to 7334146709345091584. Current Snapshot: MvccSnapshot[applied={T|T < 7334146709478207488}]
I20260928 02:39:21.496919 17767 tablet_replica.cc:439] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc36762ef32440e48126c12cf1f3d255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } }
I20260928 02:39:21.496997 17767 tablet_replica.cc:442] T 5f213afa494f45a79e034dd952c06101 P 8e0dbbf30e024ddb978e257ee5e9d1d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:39:21.497041 17765 tablet_replica.cc:439] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc36762ef32440e48126c12cf1f3d255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } }
I20260928 02:39:21.497113 17765 tablet_replica.cc:442] T 5f213afa494f45a79e034dd952c06101 P 2c99680fca5a49f6a3b355ee0c0b853b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:39:21.496441 17804 tablet_replica.cc:439] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc36762ef32440e48126c12cf1f3d255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } } }
I20260928 02:39:21.497222 17804 tablet_replica.cc:442] T 5f213afa494f45a79e034dd952c06101 P fc36762ef32440e48126c12cf1f3d255: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:39:21.502966 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.500306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:40555 (local address 127.9.130.1:46669)
0928 02:39:21.500347 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.500349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.500478 (+ 129us) server_negotiation.cc:407] Connection header received
0928 02:39:21.500533 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.500534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.500555 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.500603 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:39:21.500711 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.501240 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.502331 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.502521 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.502582 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.502758 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:39:21.502783 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:39:21.502825 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.502905 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.502920 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:39:21.503188 17807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.500227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46669 (local address 127.9.130.2:40555)
0928 02:39:21.500405 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:21.500416 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:21.500476 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:21.500607 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:21.500607 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:21.500609 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:39:21.500653 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.500655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.501321 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.501322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.502247 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.502250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.502604 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.502605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.502663 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.502677 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:39:21.502846 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:39:21.502849 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:21.502903 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:39:21.502950 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260928 02:39:21.505079 17403 raft_consensus.cc:1261] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Refusing update from remote peer 8e0dbbf30e024ddb978e257ee5e9d1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:21.505208 17683 raft_consensus.cc:1261] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Refusing update from remote peer 8e0dbbf30e024ddb978e257ee5e9d1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:21.505290 17761 consensus_queue.cc:1035] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:21.505384 17767 consensus_queue.cc:1035] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:21.514076 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.511245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:45615 (local address 127.9.130.2:43423)
0928 02:39:21.511285 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.511288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.511448 (+ 160us) server_negotiation.cc:407] Connection header received
0928 02:39:21.511541 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.511541 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.511554 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.511616 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:39:21.511773 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.512311 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.513286 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.513534 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.513620 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.513805 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:39:21.513843 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:39:21.513897 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.513966 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.513994 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:39:21.514118 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.511195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:43423 (local address 127.9.130.1:45615)
0928 02:39:21.511367 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:21.511384 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:21.511466 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:21.511619 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:21.511620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:21.511622 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:39:21.511692 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.511694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.512411 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.512413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.513184 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:21.513187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.513618 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.513619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:21.513685 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.513708 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:39:21.513903 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:39:21.513905 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:39:21.513962 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:39:21.514006 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260928 02:39:21.523018 17246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:21.523128 17246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:21.523711 17246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62: Sending DeleteTablet for 3 replicas of tablet 0aeb8457f94c4e84ace17aec180cabf2
I20260928 02:39:21.523950 17383 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aeb8457f94c4e84ace17aec180cabf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:21 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:47120
I20260928 02:39:21.523986 17663 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aeb8457f94c4e84ace17aec180cabf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:21 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:37174
I20260928 02:39:21.524006 17523 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aeb8457f94c4e84ace17aec180cabf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:21 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:59000
I20260928 02:39:21.524084 17779 tablet_replica.cc:330] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255: stopping tablet replica
I20260928 02:39:21.524081 17780 tablet_replica.cc:330] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2: stopping tablet replica
I20260928 02:39:21.524135 17779 raft_consensus.cc:2229] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:21.524140 17780 raft_consensus.cc:2229] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:39:21.524178 17779 raft_consensus.cc:2258] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:21.524192 17780 raft_consensus.cc:2258] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:21.524242 17781 tablet_replica.cc:330] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b: stopping tablet replica
I20260928 02:39:21.524292 17781 raft_consensus.cc:2229] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:21.524328 17781 raft_consensus.cc:2258] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:21.524477 17780 ts_tablet_manager.cc:1905] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:21.524710 17781 ts_tablet_manager.cc:1905] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:21.524770 17779 ts_tablet_manager.cc:1905] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:21.528064 17779 ts_tablet_manager.cc:1918] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:39:21.528120 17779 log.cc:1196] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/wal/wals/0aeb8457f94c4e84ace17aec180cabf2
I20260928 02:39:21.528332 17779 ts_tablet_manager.cc:1939] T 0aeb8457f94c4e84ace17aec180cabf2 P fc36762ef32440e48126c12cf1f3d255: Deleting consensus metadata
I20260928 02:39:21.528622 17232 catalog_manager.cc:4841] TS fc36762ef32440e48126c12cf1f3d255 (127.9.130.1:46669): tablet 0aeb8457f94c4e84ace17aec180cabf2 (table test-table [id=bffc3f91c0984a35a06bc02d912a598e]) successfully deleted
I20260928 02:39:21.528699 17781 ts_tablet_manager.cc:1918] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:39:21.528748 17781 log.cc:1196] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/0aeb8457f94c4e84ace17aec180cabf2
I20260928 02:39:21.528944 17781 ts_tablet_manager.cc:1939] T 0aeb8457f94c4e84ace17aec180cabf2 P 2c99680fca5a49f6a3b355ee0c0b853b: Deleting consensus metadata
I20260928 02:39:21.529233 17230 catalog_manager.cc:4841] TS 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961): tablet 0aeb8457f94c4e84ace17aec180cabf2 (table test-table [id=bffc3f91c0984a35a06bc02d912a598e]) successfully deleted
I20260928 02:39:21.529311 17780 ts_tablet_manager.cc:1918] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:39:21.529358 17780 log.cc:1196] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/0aeb8457f94c4e84ace17aec180cabf2
I20260928 02:39:21.529551 17780 ts_tablet_manager.cc:1939] T 0aeb8457f94c4e84ace17aec180cabf2 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Deleting consensus metadata
I20260928 02:39:21.529842 17230 catalog_manager.cc:4841] TS 8e0dbbf30e024ddb978e257ee5e9d1d2 (127.9.130.2:43423): tablet 0aeb8457f94c4e84ace17aec180cabf2 (table test-table [id=bffc3f91c0984a35a06bc02d912a598e]) successfully deleted
Sep 28 02:39:21 dist-test-slave-kkrh krb5kdc[17201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:39:21 dist-test-slave-kkrh krb5kdc[17201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:21 dist-test-slave-kkrh krb5kdc[17201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:39:21.548261 17297 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:21.541946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.9.130.62:38137)
0928 02:39:21.542009 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.542011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.542099 (+ 88us) server_negotiation.cc:407] Connection header received
0928 02:39:21.542180 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.542180 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.542189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.542238 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:21.542964 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.543579 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.544679 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.544861 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.544920 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.548184 (+ 3264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60208: BlockingRecv error: recv got EOF from 127.0.0.1:60208 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 02:39:21 dist-test-slave-kkrh krb5kdc[17201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM
I20260928 02:39:21.559279 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.549236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.9.130.62:38137)
0928 02:39:21.549273 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.549275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.551985 (+ 2710us) server_negotiation.cc:407] Connection header received
0928 02:39:21.552062 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.552062 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.552071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.552163 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:21.552722 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.553196 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.554169 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.554341 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.554388 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.556809 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:21.556810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:21.556811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:21.556825 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:21.558157 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:21.558509 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:21.558510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:21.558511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:21.558565 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:21.558755 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:21.558756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:21.558756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:21.558863 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:21.558965 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.559069 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.559177 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:39:21.560747 17246 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:39:21.560916 17246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:39:21.562232 17383 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8b37e67c8f49b09beb39d38f61a075 (DEFAULT_TABLE table=test-table [id=3318f6e2b16b495faab76119a9696286]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:21.562261 17523 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8b37e67c8f49b09beb39d38f61a075 (DEFAULT_TABLE table=test-table [id=3318f6e2b16b495faab76119a9696286]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:21.562270 17663 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8b37e67c8f49b09beb39d38f61a075 (DEFAULT_TABLE table=test-table [id=3318f6e2b16b495faab76119a9696286]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:21.562361 17523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c8b37e67c8f49b09beb39d38f61a075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.562361 17383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c8b37e67c8f49b09beb39d38f61a075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.562395 17663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c8b37e67c8f49b09beb39d38f61a075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.563606 17758 tablet_bootstrap.cc:492] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255: Bootstrap starting.
I20260928 02:39:21.563606 17757 tablet_bootstrap.cc:492] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Bootstrap starting.
I20260928 02:39:21.563719 17762 tablet_bootstrap.cc:492] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b: Bootstrap starting.
I20260928 02:39:21.564002 17757 tablet_bootstrap.cc:654] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.564008 17758 tablet_bootstrap.cc:654] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.564422 17762 tablet_bootstrap.cc:654] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.564689 17758 tablet_bootstrap.cc:492] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255: No bootstrap required, opened a new log
I20260928 02:39:21.564738 17758 ts_tablet_manager.cc:1397] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:21.564760 17757 tablet_bootstrap.cc:492] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2: No bootstrap required, opened a new log
I20260928 02:39:21.564801 17757 ts_tablet_manager.cc:1397] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:21.564881 17758 raft_consensus.cc:348] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.564894 17757 raft_consensus.cc:348] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.564946 17758 raft_consensus.cc:374] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.564947 17757 raft_consensus.cc:374] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.564968 17758 raft_consensus.cc:729] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc36762ef32440e48126c12cf1f3d255, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.564968 17757 raft_consensus.cc:729] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0dbbf30e024ddb978e257ee5e9d1d2, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.565011 17757 consensus_queue.cc:260] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.565011 17758 consensus_queue.cc:260] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.565090 17757 ts_tablet_manager.cc:1428] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:21.565090 17758 ts_tablet_manager.cc:1428] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:21.565546 17762 tablet_bootstrap.cc:492] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b: No bootstrap required, opened a new log
I20260928 02:39:21.565619 17762 ts_tablet_manager.cc:1397] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:39:21.565771 17762 raft_consensus.cc:348] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.565843 17762 raft_consensus.cc:374] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.565864 17762 raft_consensus.cc:729] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c99680fca5a49f6a3b355ee0c0b853b, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.565902 17762 consensus_queue.cc:260] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.566002 17762 ts_tablet_manager.cc:1428] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:21.596891 17810 raft_consensus.cc:482] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:39:21.597055 17810 raft_consensus.cc:504] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.597273 17810 leader_election.cc:288] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc36762ef32440e48126c12cf1f3d255 (127.9.130.1:46669), 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961)
I20260928 02:39:21.597453 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c8b37e67c8f49b09beb39d38f61a075" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" is_pre_election: true
I20260928 02:39:21.597455 17403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c8b37e67c8f49b09beb39d38f61a075" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc36762ef32440e48126c12cf1f3d255" is_pre_election: true
I20260928 02:39:21.597570 17403 raft_consensus.cc:2454] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 0.
I20260928 02:39:21.597572 17683 raft_consensus.cc:2454] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 0.
I20260928 02:39:21.597738 17476 leader_election.cc:302] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0dbbf30e024ddb978e257ee5e9d1d2, fc36762ef32440e48126c12cf1f3d255; no voters:
I20260928 02:39:21.597839 17810 raft_consensus.cc:2781] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:39:21.597895 17810 raft_consensus.cc:482] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:39:21.597926 17810 raft_consensus.cc:3037] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.598670 17810 raft_consensus.cc:504] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.598845 17810 leader_election.cc:288] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 election: Requested vote from peers fc36762ef32440e48126c12cf1f3d255 (127.9.130.1:46669), 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961)
I20260928 02:39:21.599031 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c8b37e67c8f49b09beb39d38f61a075" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c99680fca5a49f6a3b355ee0c0b853b"
I20260928 02:39:21.599022 17403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c8b37e67c8f49b09beb39d38f61a075" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc36762ef32440e48126c12cf1f3d255"
I20260928 02:39:21.599128 17403 raft_consensus.cc:3037] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.599141 17683 raft_consensus.cc:3037] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.599754 17403 raft_consensus.cc:2454] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 1.
I20260928 02:39:21.599956 17683 raft_consensus.cc:2454] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 1.
I20260928 02:39:21.600106 17476 leader_election.cc:302] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0dbbf30e024ddb978e257ee5e9d1d2, fc36762ef32440e48126c12cf1f3d255; no voters:
I20260928 02:39:21.600211 17810 raft_consensus.cc:2781] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:21.600261 17810 raft_consensus.cc:686] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 LEADER]: Becoming Leader. State: Replica: 8e0dbbf30e024ddb978e257ee5e9d1d2, State: Running, Role: LEADER
I20260928 02:39:21.600313 17810 consensus_queue.cc:237] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.600909 17245 catalog_manager.cc:5495] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0dbbf30e024ddb978e257ee5e9d1d2 (127.9.130.2). New cstate: current_term: 1 leader_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } health_report { overall_health: UNKNOWN } } }
I20260928 02:39:21.613490 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:21.611125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.9.130.2:43423)
0928 02:39:21.611187 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:21.611189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:21.611198 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:39:21.611260 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:21.611261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:21.611274 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:21.611351 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:39:21.611497 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.612040 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.612785 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:21.612976 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:21.613055 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:21.613219 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:39:21.613253 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:39:21.613327 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:21.613390 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 02:39:21.613407 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:39:21.614151 17403 raft_consensus.cc:1261] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Refusing update from remote peer 8e0dbbf30e024ddb978e257ee5e9d1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:21.614178 17683 raft_consensus.cc:1261] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Refusing update from remote peer 8e0dbbf30e024ddb978e257ee5e9d1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:21.614379 17810 consensus_queue.cc:1035] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:21.614464 17767 consensus_queue.cc:1035] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:21.616605 17245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:21.616673 17245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:21.617219 17245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62: Sending DeleteTablet for 3 replicas of tablet 5c8b37e67c8f49b09beb39d38f61a075
I20260928 02:39:21.617483 17523 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8b37e67c8f49b09beb39d38f61a075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:21 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:59000
I20260928 02:39:21.617545 17383 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8b37e67c8f49b09beb39d38f61a075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:21 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:47120
I20260928 02:39:21.617547 17663 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8b37e67c8f49b09beb39d38f61a075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:21 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:37174
I20260928 02:39:21.617592 17780 tablet_replica.cc:330] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2: stopping tablet replica
I20260928 02:39:21.617642 17781 tablet_replica.cc:330] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b: stopping tablet replica
I20260928 02:39:21.617650 17780 raft_consensus.cc:2229] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:39:21.617681 17781 raft_consensus.cc:2229] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:21.617705 17780 raft_consensus.cc:2258] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:21.617713 17781 raft_consensus.cc:2258] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:21.617782 17779 tablet_replica.cc:330] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255: stopping tablet replica
I20260928 02:39:21.617838 17779 raft_consensus.cc:2229] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:21.617874 17779 raft_consensus.cc:2258] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:21.617980 17781 ts_tablet_manager.cc:1905] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:21.618120 17780 ts_tablet_manager.cc:1905] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:21.619585 17780 ts_tablet_manager.cc:1918] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:21.619614 17781 ts_tablet_manager.cc:1918] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:21.619642 17780 log.cc:1196] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/5c8b37e67c8f49b09beb39d38f61a075
I20260928 02:39:21.619661 17781 log.cc:1196] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/5c8b37e67c8f49b09beb39d38f61a075
I20260928 02:39:21.619626 17245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:39:21.619782 17245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:39:21.619786 17779 ts_tablet_manager.cc:1905] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:21.619884 17780 ts_tablet_manager.cc:1939] T 5c8b37e67c8f49b09beb39d38f61a075 P 8e0dbbf30e024ddb978e257ee5e9d1d2: Deleting consensus metadata
I20260928 02:39:21.619886 17781 ts_tablet_manager.cc:1939] T 5c8b37e67c8f49b09beb39d38f61a075 P 2c99680fca5a49f6a3b355ee0c0b853b: Deleting consensus metadata
I20260928 02:39:21.620191 17230 catalog_manager.cc:4841] TS 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961): tablet 5c8b37e67c8f49b09beb39d38f61a075 (table test-table [id=3318f6e2b16b495faab76119a9696286]) successfully deleted
I20260928 02:39:21.620251 17230 catalog_manager.cc:4841] TS 8e0dbbf30e024ddb978e257ee5e9d1d2 (127.9.130.2:43423): tablet 5c8b37e67c8f49b09beb39d38f61a075 (table test-table [id=3318f6e2b16b495faab76119a9696286]) successfully deleted
I20260928 02:39:21.621132 17779 ts_tablet_manager.cc:1918] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:21.621153 17523 tablet_service.cc:1467] Processing CreateTablet for tablet 138a84746fda420fb00f7f96b303a1df (DEFAULT_TABLE table=test-table [id=b2ac6c92fc394d6fa5108f8604dcb2b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:21.621165 17663 tablet_service.cc:1467] Processing CreateTablet for tablet 138a84746fda420fb00f7f96b303a1df (DEFAULT_TABLE table=test-table [id=b2ac6c92fc394d6fa5108f8604dcb2b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:21.621202 17779 log.cc:1196] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/wal/wals/5c8b37e67c8f49b09beb39d38f61a075
I20260928 02:39:21.621244 17523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138a84746fda420fb00f7f96b303a1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.621264 17663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138a84746fda420fb00f7f96b303a1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.621423 17779 ts_tablet_manager.cc:1939] T 5c8b37e67c8f49b09beb39d38f61a075 P fc36762ef32440e48126c12cf1f3d255: Deleting consensus metadata
I20260928 02:39:21.621807 17232 catalog_manager.cc:4841] TS fc36762ef32440e48126c12cf1f3d255 (127.9.130.1:46669): tablet 5c8b37e67c8f49b09beb39d38f61a075 (table test-table [id=3318f6e2b16b495faab76119a9696286]) successfully deleted
I20260928 02:39:21.622227 17762 tablet_bootstrap.cc:492] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b: Bootstrap starting.
I20260928 02:39:21.622268 17383 tablet_service.cc:1467] Processing CreateTablet for tablet 138a84746fda420fb00f7f96b303a1df (DEFAULT_TABLE table=test-table [id=b2ac6c92fc394d6fa5108f8604dcb2b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:21.622351 17383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138a84746fda420fb00f7f96b303a1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:21.622503 17757 tablet_bootstrap.cc:492] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2: Bootstrap starting.
I20260928 02:39:21.622620 17762 tablet_bootstrap.cc:654] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.622750 17757 tablet_bootstrap.cc:654] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.623211 17757 tablet_bootstrap.cc:492] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2: No bootstrap required, opened a new log
I20260928 02:39:21.623250 17757 ts_tablet_manager.cc:1397] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:39:21.623322 17757 raft_consensus.cc:348] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.623368 17757 raft_consensus.cc:374] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.623380 17757 raft_consensus.cc:729] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0dbbf30e024ddb978e257ee5e9d1d2, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.623409 17757 consensus_queue.cc:260] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.623466 17757 ts_tablet_manager.cc:1428] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:21.623529 17758 tablet_bootstrap.cc:492] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255: Bootstrap starting.
I20260928 02:39:21.623848 17758 tablet_bootstrap.cc:654] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:21.624209 17762 tablet_bootstrap.cc:492] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b: No bootstrap required, opened a new log
I20260928 02:39:21.624259 17762 ts_tablet_manager.cc:1397] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:39:21.624378 17758 tablet_bootstrap.cc:492] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255: No bootstrap required, opened a new log
I20260928 02:39:21.624428 17758 ts_tablet_manager.cc:1397] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:21.624393 17762 raft_consensus.cc:348] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.624500 17762 raft_consensus.cc:374] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.624528 17762 raft_consensus.cc:729] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c99680fca5a49f6a3b355ee0c0b853b, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.624536 17758 raft_consensus.cc:348] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.624577 17762 consensus_queue.cc:260] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.624601 17758 raft_consensus.cc:374] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:21.624629 17758 raft_consensus.cc:729] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc36762ef32440e48126c12cf1f3d255, State: Initialized, Role: FOLLOWER
I20260928 02:39:21.624670 17762 ts_tablet_manager.cc:1428] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:21.624691 17758 consensus_queue.cc:260] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.624785 17758 ts_tablet_manager.cc:1428] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:21.943527 17761 raft_consensus.cc:482] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:39:21.943617 17761 raft_consensus.cc:504] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.943799 17761 leader_election.cc:288] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc36762ef32440e48126c12cf1f3d255 (127.9.130.1:46669), 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961)
I20260928 02:39:21.944087 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138a84746fda420fb00f7f96b303a1df" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" is_pre_election: true
I20260928 02:39:21.944083 17403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138a84746fda420fb00f7f96b303a1df" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc36762ef32440e48126c12cf1f3d255" is_pre_election: true
I20260928 02:39:21.944173 17683 raft_consensus.cc:2454] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 0.
I20260928 02:39:21.944191 17403 raft_consensus.cc:2454] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 0.
I20260928 02:39:21.944352 17474 leader_election.cc:302] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c99680fca5a49f6a3b355ee0c0b853b, 8e0dbbf30e024ddb978e257ee5e9d1d2; no voters:
I20260928 02:39:21.944466 17761 raft_consensus.cc:2781] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:39:21.944516 17761 raft_consensus.cc:482] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:39:21.944531 17761 raft_consensus.cc:3037] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.945134 17761 raft_consensus.cc:504] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.945273 17761 leader_election.cc:288] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 election: Requested vote from peers fc36762ef32440e48126c12cf1f3d255 (127.9.130.1:46669), 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961)
I20260928 02:39:21.945446 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138a84746fda420fb00f7f96b303a1df" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c99680fca5a49f6a3b355ee0c0b853b"
I20260928 02:39:21.945429 17403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138a84746fda420fb00f7f96b303a1df" candidate_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc36762ef32440e48126c12cf1f3d255"
I20260928 02:39:21.945514 17403 raft_consensus.cc:3037] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.945536 17683 raft_consensus.cc:3037] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:21.946084 17683 raft_consensus.cc:2454] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 1.
I20260928 02:39:21.946156 17403 raft_consensus.cc:2454] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0dbbf30e024ddb978e257ee5e9d1d2 in term 1.
I20260928 02:39:21.946267 17474 leader_election.cc:302] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c99680fca5a49f6a3b355ee0c0b853b, 8e0dbbf30e024ddb978e257ee5e9d1d2; no voters:
I20260928 02:39:21.946352 17761 raft_consensus.cc:2781] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:21.946413 17761 raft_consensus.cc:686] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 LEADER]: Becoming Leader. State: Replica: 8e0dbbf30e024ddb978e257ee5e9d1d2, State: Running, Role: LEADER
I20260928 02:39:21.946470 17761 consensus_queue.cc:237] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } }
I20260928 02:39:21.947032 17245 catalog_manager.cc:5495] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0dbbf30e024ddb978e257ee5e9d1d2 (127.9.130.2). New cstate: current_term: 1 leader_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0dbbf30e024ddb978e257ee5e9d1d2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 43423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 } health_report { overall_health: UNKNOWN } } }
I20260928 02:39:21.979430 17683 raft_consensus.cc:1261] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Refusing update from remote peer 8e0dbbf30e024ddb978e257ee5e9d1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:21.979430 17403 raft_consensus.cc:1261] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Refusing update from remote peer 8e0dbbf30e024ddb978e257ee5e9d1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:21.979696 17761 consensus_queue.cc:1035] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c99680fca5a49f6a3b355ee0c0b853b" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:21.979822 17810 consensus_queue.cc:1035] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc36762ef32440e48126c12cf1f3d255" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:21.990012 17245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:21.990125 17245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:21.990764 17245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9937968dbb6b4fda88022d40089c2d62: Sending DeleteTablet for 3 replicas of tablet 138a84746fda420fb00f7f96b303a1df
I20260928 02:39:21.991073 17383 tablet_service.cc:1514] Processing DeleteTablet for tablet 138a84746fda420fb00f7f96b303a1df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:21 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:47120
I20260928 02:39:21.991079 17523 tablet_service.cc:1514] Processing DeleteTablet for tablet 138a84746fda420fb00f7f96b303a1df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:21 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:59000
I20260928 02:39:21.991106 17663 tablet_service.cc:1514] Processing DeleteTablet for tablet 138a84746fda420fb00f7f96b303a1df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:21 UTC) from {username='oryx', principal='oryx/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:37174
I20260928 02:39:21.991181 17779 tablet_replica.cc:330] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255: stopping tablet replica
I20260928 02:39:21.991195 17781 tablet_replica.cc:330] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b: stopping tablet replica
I20260928 02:39:21.991237 17781 raft_consensus.cc:2229] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:21.991238 17779 raft_consensus.cc:2229] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:21.991276 17781 raft_consensus.cc:2258] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:21.991276 17779 raft_consensus.cc:2258] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:21.991351 17780 tablet_replica.cc:330] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2: stopping tablet replica
I20260928 02:39:21.991415 17780 raft_consensus.cc:2229] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:39:21.991461 17780 raft_consensus.cc:2258] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:21.991775 17780 ts_tablet_manager.cc:1905] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:21.991765 17781 ts_tablet_manager.cc:1905] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:21.991952 17779 ts_tablet_manager.cc:1905] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:21.993278 17780 ts_tablet_manager.cc:1918] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:39:21.993319 17781 ts_tablet_manager.cc:1918] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:39:21.993350 17780 log.cc:1196] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/138a84746fda420fb00f7f96b303a1df
I20260928 02:39:21.993383 17781 log.cc:1196] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/138a84746fda420fb00f7f96b303a1df
I20260928 02:39:21.993394 17779 ts_tablet_manager.cc:1918] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:39:21.993449 17779 log.cc:1196] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563127283700-9736-0/minicluster-data/ts-0/wal/wals/138a84746fda420fb00f7f96b303a1df
I20260928 02:39:21.993583 17780 ts_tablet_manager.cc:1939] T 138a84746fda420fb00f7f96b303a1df P 8e0dbbf30e024ddb978e257ee5e9d1d2: Deleting consensus metadata
I20260928 02:39:21.993598 17781 ts_tablet_manager.cc:1939] T 138a84746fda420fb00f7f96b303a1df P 2c99680fca5a49f6a3b355ee0c0b853b: Deleting consensus metadata
I20260928 02:39:21.993670 17779 ts_tablet_manager.cc:1939] T 138a84746fda420fb00f7f96b303a1df P fc36762ef32440e48126c12cf1f3d255: Deleting consensus metadata
I20260928 02:39:21.993932 17230 catalog_manager.cc:4841] TS 8e0dbbf30e024ddb978e257ee5e9d1d2 (127.9.130.2:43423): tablet 138a84746fda420fb00f7f96b303a1df (table test-table [id=b2ac6c92fc394d6fa5108f8604dcb2b2]) successfully deleted
I20260928 02:39:21.993937 17232 catalog_manager.cc:4841] TS fc36762ef32440e48126c12cf1f3d255 (127.9.130.1:46669): tablet 138a84746fda420fb00f7f96b303a1df (table test-table [id=b2ac6c92fc394d6fa5108f8604dcb2b2]) successfully deleted
I20260928 02:39:21.994036 17230 catalog_manager.cc:4841] TS 2c99680fca5a49f6a3b355ee0c0b853b (127.9.130.3:32961): tablet 138a84746fda420fb00f7f96b303a1df (table test-table [id=b2ac6c92fc394d6fa5108f8604dcb2b2]) successfully deleted
I20260928 02:39:21.994263 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 17318
I20260928 02:39:22.000375 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 17458
I20260928 02:39:22.005982 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 17598
I20260928 02:39:22.010308 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 17214
2026-09-28T02:39:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2729 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:39:22 dist-test-slave-kkrh krb5kdc[17841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:39:22 dist-test-slave-kkrh krb5kdc[17841](info): set up 2 sockets
Sep 28 02:39:22 dist-test-slave-kkrh krb5kdc[17841](info): commencing operation
krb5kdc: starting...
W20260928 02:39:23.136343 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:39:23 dist-test-slave-kkrh krb5kdc[17841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:39:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:39:23.253074 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:33211
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:33211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:23.333657 17854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:23.333855 17854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:23.333885 17854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:23.335081 17854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:23.335121 17854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:23.335134 17854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:23.335147 17854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:23.335160 17854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:23.336678 17854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:33211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:33211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:23.336930 17854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:23.337179 17854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:23.342257 17862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:23.342272 17860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:23.342284 17859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:23.342629 17854 server_base.cc:1034] running on GCE node
I20260928 02:39:23.342928 17854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:23.343269 17854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:23.344449 17854 hybrid_clock.cc:648] HybridClock initialized: now 1790563163344442 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T02:39:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1339 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:39:23 dist-test-slave-kkrh krb5kdc[17867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:39:23 dist-test-slave-kkrh krb5kdc[17867](info): set up 2 sockets
Sep 28 02:39:23 dist-test-slave-kkrh krb5kdc[17867](info): commencing operation
krb5kdc: starting...
W20260928 02:39:24.400674 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:39:24 dist-test-slave-kkrh krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:39:24.528520 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:34689
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:34689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:24.609038 17880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:24.609234 17880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:24.609256 17880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:24.610301 17880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:24.610334 17880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:24.610348 17880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:24.610360 17880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:24.610373 17880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:24.611708 17880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:34689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:34689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:24.611948 17880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:24.612121 17880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:24.616788 17885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:24.616799 17886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:24.616799 17888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:24.617107 17880 server_base.cc:1034] running on GCE node
I20260928 02:39:24.617326 17880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:24.617573 17880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:24.618734 17880 hybrid_clock.cc:648] HybridClock initialized: now 1790563164618721 us; error 26 us; skew 500 ppm
Sep 28 02:39:24 dist-test-slave-kkrh krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563164, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:24.624331 17880 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:39:24.624811 17880 webserver.cc:460] Webserver started at http://127.9.130.62:39567/ using document root <none> and password file <none>
I20260928 02:39:24.625062 17880 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:24.625131 17880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:24.625283 17880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:24.626338 17880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "f5109056954245bdb008fa9e3d30b1fa"
format_stamp: "Formatted at 2026-09-28 02:39:24 on dist-test-slave-kkrh"
server_key: "131218ec8a9fee1e5765cc57532e9954"
server_key_iv: "bc33a9b4fa839cdfba9e085340e3bac2"
server_key_version: "encryptionkey@0"
I20260928 02:39:24.626722 17880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "f5109056954245bdb008fa9e3d30b1fa"
format_stamp: "Formatted at 2026-09-28 02:39:24 on dist-test-slave-kkrh"
server_key: "131218ec8a9fee1e5765cc57532e9954"
server_key_iv: "bc33a9b4fa839cdfba9e085340e3bac2"
server_key_version: "encryptionkey@0"
I20260928 02:39:24.628168 17880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:39:24.629029 17895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:24.629242 17880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:24.629347 17880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "f5109056954245bdb008fa9e3d30b1fa"
format_stamp: "Formatted at 2026-09-28 02:39:24 on dist-test-slave-kkrh"
server_key: "131218ec8a9fee1e5765cc57532e9954"
server_key_iv: "bc33a9b4fa839cdfba9e085340e3bac2"
server_key_version: "encryptionkey@0"
I20260928 02:39:24.629446 17880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:24.661074 17880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:24.663450 17880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:24.663614 17880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:24.667644 17880 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:34689
I20260928 02:39:24.667685 17947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:34689 every 8 connection(s)
I20260928 02:39:24.668085 17880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:24.669056 17948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:24.671404 17948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa: Bootstrap starting.
I20260928 02:39:24.672024 17948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:24.672298 17948 log.cc:824] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:24.672959 17948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa: No bootstrap required, opened a new log
I20260928 02:39:24.673417 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 17880
I20260928 02:39:24.673532 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:24.673729 9736 external_mini_cluster.cc:1428] Setting key 393832c6a0b5c4347d4fe67d7904b37e
I20260928 02:39:24.675012 17948 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5109056954245bdb008fa9e3d30b1fa" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34689 } }
I20260928 02:39:24.675189 17948 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:24.675228 17948 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5109056954245bdb008fa9e3d30b1fa, State: Initialized, Role: FOLLOWER
I20260928 02:39:24.675369 17948 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5109056954245bdb008fa9e3d30b1fa" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34689 } }
I20260928 02:39:24.675457 17948 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:24.675488 17948 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:24.675549 17948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:24.676491 17948 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5109056954245bdb008fa9e3d30b1fa" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34689 } }
I20260928 02:39:24.676615 17948 leader_election.cc:302] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5109056954245bdb008fa9e3d30b1fa; no voters:
I20260928 02:39:24.676781 17948 leader_election.cc:288] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:24.676819 17953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:24.676976 17953 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 1 LEADER]: Becoming Leader. State: Replica: f5109056954245bdb008fa9e3d30b1fa, State: Running, Role: LEADER
I20260928 02:39:24.677016 17948 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:24.677124 17953 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5109056954245bdb008fa9e3d30b1fa" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34689 } }
I20260928 02:39:24.677697 17955 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5109056954245bdb008fa9e3d30b1fa. Latest consensus state: current_term: 1 leader_uuid: "f5109056954245bdb008fa9e3d30b1fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5109056954245bdb008fa9e3d30b1fa" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34689 } } }
I20260928 02:39:24.677768 17955 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:24.678056 17954 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5109056954245bdb008fa9e3d30b1fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5109056954245bdb008fa9e3d30b1fa" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34689 } } }
I20260928 02:39:24.678119 17954 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:24.678335 17962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:24.678885 17962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:24.680473 17962 catalog_manager.cc:1347] Generated new cluster ID: d58f9c08806a49978ecd599fcc3d8fb7
I20260928 02:39:24.680523 17962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:39:24 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:24.685479 17952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:24.674301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.9.130.62:34689)
0928 02:39:24.674469 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:24.674478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:24.674512 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:39:24.674991 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:24.674996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:24.675041 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:24.675161 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:24.675984 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:24.677070 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:24.678163 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:24.678780 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:24.678897 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:24.681716 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:24.681720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:24.681720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:24.681732 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:24.683680 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:24.684080 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:24.684083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:24.684083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:24.684213 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:24.684420 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:24.684421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:24.684421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:24.684740 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:24.684896 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:24.684970 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:39:24.685114 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260928 02:39:24.698423 17962 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:24.698911 17962 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:24.706743 17962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa: Generated new TSK 0
I20260928 02:39:24.706941 17962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:39:24.737103 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34689
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:24.816947 17974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:24.817144 17974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:24.817167 17974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:24.818308 17974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:24.818349 17974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:24.818375 17974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:24.819763 17974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:24.820015 17974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:24.820236 17974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:24.825366 17980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:24.825349 17982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:24.825570 17979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:24.825662 17974 server_base.cc:1034] running on GCE node
I20260928 02:39:24.825829 17974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:24.826027 17974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:24.827198 17974 hybrid_clock.cc:648] HybridClock initialized: now 1790563164827174 us; error 34 us; skew 500 ppm
Sep 28 02:39:24 dist-test-slave-kkrh krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563164, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:24.829974 17974 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:39:24.830418 17974 webserver.cc:460] Webserver started at http://127.9.130.1:45525/ using document root <none> and password file <none>
I20260928 02:39:24.830665 17974 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:24.830736 17974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:24.830885 17974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:24.832051 17974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "8c811e40cc874533abb5e35157d1d1b2"
format_stamp: "Formatted at 2026-09-28 02:39:24 on dist-test-slave-kkrh"
server_key: "a6a823e558609d347bc547636502dac0"
server_key_iv: "b5877fff319d809be478f5f20f58a74e"
server_key_version: "encryptionkey@0"
I20260928 02:39:24.832460 17974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "8c811e40cc874533abb5e35157d1d1b2"
format_stamp: "Formatted at 2026-09-28 02:39:24 on dist-test-slave-kkrh"
server_key: "a6a823e558609d347bc547636502dac0"
server_key_iv: "b5877fff319d809be478f5f20f58a74e"
server_key_version: "encryptionkey@0"
I20260928 02:39:24.833849 17974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 02:39:24.834661 17989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:24.834863 17974 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:24.834969 17974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "8c811e40cc874533abb5e35157d1d1b2"
format_stamp: "Formatted at 2026-09-28 02:39:24 on dist-test-slave-kkrh"
server_key: "a6a823e558609d347bc547636502dac0"
server_key_iv: "b5877fff319d809be478f5f20f58a74e"
server_key_version: "encryptionkey@0"
I20260928 02:39:24.835050 17974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:24.848783 17974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:24.851084 17974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:24.851244 17974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:24.851500 17974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:24.851879 17974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:24.851939 17974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:24.851984 17974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:24.852010 17974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:24.857605 17974 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:32931
I20260928 02:39:24.857654 18102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:32931 every 8 connection(s)
I20260928 02:39:24.857951 17974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:39:24.861661 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 17974
I20260928 02:39:24.861811 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:39:24.862059 9736 external_mini_cluster.cc:1428] Setting key 8c8209cf724ab71e51ef6d494f28f0ea
Sep 28 02:39:24 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563164, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:24.869165 17952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:24.858653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:47877 (local address 127.9.130.62:34689)
0928 02:39:24.858730 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:24.858732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:24.858948 (+ 216us) server_negotiation.cc:407] Connection header received
0928 02:39:24.859488 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:24.859489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:24.859507 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:24.859569 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:24.860727 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:24.861231 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:24.862494 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:24.862831 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:24.862919 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:24.865799 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:24.865801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:24.865801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:24.865811 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:24.867406 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:24.868003 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:24.868004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:24.868004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:24.868061 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:24.868387 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:24.868388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:24.868388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:24.868500 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:24.868552 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:24.868966 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 02:39:24.869094 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:39:24.869410 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:24.858646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:47877)
0928 02:39:24.858822 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:24.858855 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:24.859326 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:24.859677 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:24.859681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:24.860405 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:24.860625 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:24.860630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:24.861322 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:24.861323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:24.862398 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:24.862401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:24.862942 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:24.862944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:24.863120 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:24.863703 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:24.863714 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:24.865704 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:24.867505 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:24.867512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:24.867514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:24.867906 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:24.868142 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:24.868143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:24.868144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:24.868311 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:24.868580 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:24.868586 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:24.868791 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 02:39:24.868921 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 02:39:24.869140 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260928 02:39:24.870033 18103 heartbeater.cc:347] Connected to a master server at 127.9.130.62:34689
I20260928 02:39:24.870155 18103 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:24.870347 18103 heartbeater.cc:511] Master 127.9.130.62:34689 requested a full tablet report, sending...
I20260928 02:39:24.870939 17912 ts_manager.cc:194] Registered new tserver with Master: 8c811e40cc874533abb5e35157d1d1b2 (127.9.130.1:32931)
I20260928 02:39:24.871625 17912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:47877
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:39:24.890236 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34689
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:24.975865 18108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:24.976071 18108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:24.976095 18108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:24.977294 18108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:24.977334 18108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:24.977360 18108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:24.978819 18108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:24.978992 18108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:24.979173 18108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:24.984105 18113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:24.984076 18114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:24.984097 18116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:24.984366 18108 server_base.cc:1034] running on GCE node
I20260928 02:39:24.984625 18108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:24.984854 18108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:24.986027 18108 hybrid_clock.cc:648] HybridClock initialized: now 1790563164986021 us; error 36 us; skew 500 ppm
Sep 28 02:39:24 dist-test-slave-kkrh krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563164, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:24.988622 18108 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:39:24.989068 18108 webserver.cc:460] Webserver started at http://127.9.130.2:39321/ using document root <none> and password file <none>
I20260928 02:39:24.989295 18108 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:24.989363 18108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:24.989522 18108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:24.990587 18108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "d070d5be9c0b46f2ada0d5a6fac1934d"
format_stamp: "Formatted at 2026-09-28 02:39:24 on dist-test-slave-kkrh"
server_key: "37071c5d2dc659033020c01dda898c55"
server_key_iv: "de39a0646ff3dce95c0eb8b4fccd886b"
server_key_version: "encryptionkey@0"
I20260928 02:39:24.990988 18108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "d070d5be9c0b46f2ada0d5a6fac1934d"
format_stamp: "Formatted at 2026-09-28 02:39:24 on dist-test-slave-kkrh"
server_key: "37071c5d2dc659033020c01dda898c55"
server_key_iv: "de39a0646ff3dce95c0eb8b4fccd886b"
server_key_version: "encryptionkey@0"
I20260928 02:39:24.992447 18108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260928 02:39:24.993258 18123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:24.993484 18108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:24.993610 18108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "d070d5be9c0b46f2ada0d5a6fac1934d"
format_stamp: "Formatted at 2026-09-28 02:39:24 on dist-test-slave-kkrh"
server_key: "37071c5d2dc659033020c01dda898c55"
server_key_iv: "de39a0646ff3dce95c0eb8b4fccd886b"
server_key_version: "encryptionkey@0"
I20260928 02:39:24.993705 18108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:25.015385 18108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:25.017617 18108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:25.017784 18108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:25.018015 18108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:25.018325 18108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:25.018379 18108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.018416 18108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:25.018442 18108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.024474 18108 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:42589
I20260928 02:39:25.024520 18236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:42589 every 8 connection(s)
I20260928 02:39:25.024838 18108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:39:25.025146 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 18108
I20260928 02:39:25.025269 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:39:25.025543 9736 external_mini_cluster.cc:1428] Setting key 1d2d367707ec73291a0aea37f0a3a67f
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563164, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
I20260928 02:39:25.035794 17952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.025714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:45399 (local address 127.9.130.62:34689)
0928 02:39:25.025897 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:25.025900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:25.026113 (+ 213us) server_negotiation.cc:407] Connection header received
0928 02:39:25.026605 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:25.026606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:25.026620 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:25.026687 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:25.027715 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.028318 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.029609 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.029915 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.029989 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.032597 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:25.032599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:25.032600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:25.032608 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:25.034349 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:25.034774 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:25.034775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:25.034776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:25.034843 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:25.035109 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:25.035110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:25.035110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:25.035251 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:25.035322 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:25.035593 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 02:39:25.035686 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260928 02:39:25.035985 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.025648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:45399)
0928 02:39:25.025968 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:25.026009 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:25.026508 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:25.026765 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:25.026768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:25.027343 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:25.027543 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.027548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.028436 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.028438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.029483 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.029486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.030039 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.030040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.030149 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.030790 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:25.030799 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:25.032468 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:25.034438 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:25.034444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:25.034446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:25.034691 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:25.034924 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:25.034925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:25.034925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:25.035037 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:25.035324 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:25.035329 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:25.035459 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 02:39:25.035567 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:39:25.035701 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":89,"threads_started":1}
I20260928 02:39:25.036748 18237 heartbeater.cc:347] Connected to a master server at 127.9.130.62:34689
I20260928 02:39:25.036860 18237 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:25.037034 18237 heartbeater.cc:511] Master 127.9.130.62:34689 requested a full tablet report, sending...
I20260928 02:39:25.037438 17912 ts_manager.cc:194] Registered new tserver with Master: d070d5be9c0b46f2ada0d5a6fac1934d (127.9.130.2:42589)
I20260928 02:39:25.037870 17912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:45399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:39:25.054404 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34689
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:25.135671 18242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:25.135931 18242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:25.135970 18242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:25.137141 18242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:25.137207 18242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:25.137253 18242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:25.139458 18242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:25.139719 18242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:25.140251 18242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:25.145974 18247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:25.146011 18248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:25.145982 18250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:25.146256 18242 server_base.cc:1034] running on GCE node
I20260928 02:39:25.146449 18242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:25.146684 18242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:25.147891 18242 hybrid_clock.cc:648] HybridClock initialized: now 1790563165147878 us; error 30 us; skew 500 ppm
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563165, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:25.150559 18242 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:39:25.150965 18242 webserver.cc:460] Webserver started at http://127.9.130.3:39989/ using document root <none> and password file <none>
I20260928 02:39:25.151176 18242 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:25.151217 18242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:25.151305 18242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:25.152420 18242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "2ddfedf97f684c39b20d053e29a57c0b"
format_stamp: "Formatted at 2026-09-28 02:39:25 on dist-test-slave-kkrh"
server_key: "2f5414d4ae219c8f2686118d9258eebd"
server_key_iv: "6d466862dff70e3c2fdc21798bcbe47d"
server_key_version: "encryptionkey@0"
I20260928 02:39:25.152724 18242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "2ddfedf97f684c39b20d053e29a57c0b"
format_stamp: "Formatted at 2026-09-28 02:39:25 on dist-test-slave-kkrh"
server_key: "2f5414d4ae219c8f2686118d9258eebd"
server_key_iv: "6d466862dff70e3c2fdc21798bcbe47d"
server_key_version: "encryptionkey@0"
I20260928 02:39:25.154001 18242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:25.154690 18257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.154901 18242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:25.155009 18242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "2ddfedf97f684c39b20d053e29a57c0b"
format_stamp: "Formatted at 2026-09-28 02:39:25 on dist-test-slave-kkrh"
server_key: "2f5414d4ae219c8f2686118d9258eebd"
server_key_iv: "6d466862dff70e3c2fdc21798bcbe47d"
server_key_version: "encryptionkey@0"
I20260928 02:39:25.155092 18242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:25.177246 18242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:25.179472 18242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:25.179628 18242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:25.179900 18242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:25.180228 18242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:25.180282 18242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.180323 18242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:25.180349 18242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.186096 18242 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:35791
I20260928 02:39:25.186177 18370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:35791 every 8 connection(s)
I20260928 02:39:25.186503 18242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:39:25.189218 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 18242
I20260928 02:39:25.189296 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:39:25.189486 9736 external_mini_cluster.cc:1428] Setting key 057e3efe840bb6a50cac3ba7b872c497
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563165, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:25.195755 17952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.187261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:56403 (local address 127.9.130.62:34689)
0928 02:39:25.187306 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:25.187309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:25.187503 (+ 194us) server_negotiation.cc:407] Connection header received
0928 02:39:25.188088 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:25.188089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:25.188099 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:25.188159 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:25.189115 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.189795 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.191037 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.191240 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.191303 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.193317 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:25.193318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:25.193318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:25.193324 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:25.194510 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:25.194895 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:25.194896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:25.194896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:25.194945 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:25.195241 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:25.195241 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:25.195242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:25.195334 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:25.195383 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:25.195637 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:39:25.195693 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:39:25.196031 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.187239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:56403)
0928 02:39:25.187380 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:25.187414 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:25.187975 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:25.188215 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:25.188218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:25.188792 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:25.188987 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.188993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.189910 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.189911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.190926 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.190929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.191317 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.191317 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.191412 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.191923 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:25.191930 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:25.193238 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:25.194598 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:25.194603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:25.194605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:25.194827 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:25.195039 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:25.195040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:25.195040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:25.195174 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:25.195394 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:25.195398 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:25.195518 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 02:39:25.195611 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:39:25.195737 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260928 02:39:25.196506 18371 heartbeater.cc:347] Connected to a master server at 127.9.130.62:34689
I20260928 02:39:25.196672 18371 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:25.196857 18371 heartbeater.cc:511] Master 127.9.130.62:34689 requested a full tablet report, sending...
I20260928 02:39:25.197227 17912 ts_manager.cc:194] Registered new tserver with Master: 2ddfedf97f684c39b20d053e29a57c0b (127.9.130.3:35791)
I20260928 02:39:25.197585 17912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:56403
I20260928 02:39:25.200944 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab.
Entry for principal oryx/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab.
WARNING: no policy specified for oryx/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab.
Entry for principal oryx/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab.
WARNING: no policy specified for oryx/127.9.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab.
Entry for principal oryx/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab.
I20260928 02:39:25.271157 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 17974
I20260928 02:39:25.276889 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 18108
I20260928 02:39:25.281385 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 18242
I20260928 02:39:25.285704 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 17880
I20260928 02:39:25.290625 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:34689
--webserver_interface=127.9.130.62
--webserver_port=39567
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:34689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:25.370538 18380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:25.370747 18380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:25.370769 18380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:25.371984 18380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:25.372022 18380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:25.372037 18380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:25.372051 18380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:25.372103 18380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:25.373554 18380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:34689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:34689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=39567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:25.373809 18380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:25.374054 18380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:25.378897 18386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:25.378902 18385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:25.378901 18388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:25.378949 18380 server_base.cc:1034] running on GCE node
I20260928 02:39:25.379346 18380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:25.379674 18380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:25.380844 18380 hybrid_clock.cc:648] HybridClock initialized: now 1790563165380820 us; error 36 us; skew 500 ppm
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563165, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:25.383317 18380 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:39:25.383773 18380 webserver.cc:460] Webserver started at http://127.9.130.62:39567/ using document root <none> and password file <none>
I20260928 02:39:25.384033 18380 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:25.384123 18380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:25.385468 18380 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:25.386128 18395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.386292 18380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:25.386396 18380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "f5109056954245bdb008fa9e3d30b1fa"
format_stamp: "Formatted at 2026-09-28 02:39:24 on dist-test-slave-kkrh"
server_key: "131218ec8a9fee1e5765cc57532e9954"
server_key_iv: "bc33a9b4fa839cdfba9e085340e3bac2"
server_key_version: "encryptionkey@0"
I20260928 02:39:25.386776 18380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:25.399255 18380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:25.401185 18380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:25.401352 18380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:25.405431 18380 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:34689
I20260928 02:39:25.405467 18447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:34689 every 8 connection(s)
I20260928 02:39:25.405835 18380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:25.407226 18448 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:39:25.407802 18448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa: Bootstrap starting.
I20260928 02:39:25.415089 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 18380
I20260928 02:39:25.415387 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:32931
--local_ip_for_outbound_sockets=127.9.130.1
--tserver_master_addrs=127.9.130.62:34689
--webserver_port=45525
--webserver_interface=127.9.130.1
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.130.1
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:39:25.415684 18448 log.cc:824] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:25.417851 18448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:39:25.418174 18448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa: Bootstrap complete.
I20260928 02:39:25.420527 18448 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5109056954245bdb008fa9e3d30b1fa" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34689 } }
I20260928 02:39:25.420753 18448 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5109056954245bdb008fa9e3d30b1fa, State: Initialized, Role: FOLLOWER
I20260928 02:39:25.420922 18448 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5109056954245bdb008fa9e3d30b1fa" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34689 } }
I20260928 02:39:25.420980 18448 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:25.421042 18448 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:25.421082 18448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:39:25.422081 18448 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5109056954245bdb008fa9e3d30b1fa" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34689 } }
I20260928 02:39:25.422164 18448 leader_election.cc:302] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5109056954245bdb008fa9e3d30b1fa; no voters:
I20260928 02:39:25.422341 18448 leader_election.cc:288] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:39:25.422382 18452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:39:25.422498 18452 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [term 2 LEADER]: Becoming Leader. State: Replica: f5109056954245bdb008fa9e3d30b1fa, State: Running, Role: LEADER
I20260928 02:39:25.422580 18452 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5109056954245bdb008fa9e3d30b1fa" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34689 } }
I20260928 02:39:25.422607 18448 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:25.424137 18454 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f5109056954245bdb008fa9e3d30b1fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5109056954245bdb008fa9e3d30b1fa" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34689 } } }
I20260928 02:39:25.424237 18454 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:25.424507 18453 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5109056954245bdb008fa9e3d30b1fa. Latest consensus state: current_term: 2 leader_uuid: "f5109056954245bdb008fa9e3d30b1fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5109056954245bdb008fa9e3d30b1fa" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34689 } } }
I20260928 02:39:25.424530 18466 catalog_manager.cc:1259] Loaded cluster ID: d58f9c08806a49978ecd599fcc3d8fb7
I20260928 02:39:25.424613 18453 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:25.424619 18466 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa: loading cluster ID for follower catalog manager: success
I20260928 02:39:25.425536 18466 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa: acquiring CA information for follower catalog manager: success
I20260928 02:39:25.425920 18466 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:39:25.426000 18468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:25.426172 18468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:25.426231 18468 catalog_manager.cc:1259] Loaded cluster ID: d58f9c08806a49978ecd599fcc3d8fb7
I20260928 02:39:25.426258 18468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:39:25.426409 18468 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:25.426483 18468 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f5109056954245bdb008fa9e3d30b1fa: Loaded TSK: 0
I20260928 02:39:25.426606 18468 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:39:25.497141 18451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:25.497337 18451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:25.497360 18451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:25.498519 18451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:25.498561 18451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:25.498588 18451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:25.500056 18451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.1.keytab
--principal=oryx/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:32931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=45525
--tserver_master_addrs=127.9.130.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:25.500275 18451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:25.500489 18451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:25.506481 18474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:25.506454 18475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:25.506570 18451 server_base.cc:1034] running on GCE node
W20260928 02:39:25.506518 18477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:25.506901 18451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:25.507191 18451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:25.508378 18451 hybrid_clock.cc:648] HybridClock initialized: now 1790563165508347 us; error 51 us; skew 500 ppm
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563165, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:25.511286 18451 init.cc:377] Logged in from keytab as oryx/127.9.130.1@KRBTEST.COM (short username oryx)
I20260928 02:39:25.511821 18451 webserver.cc:460] Webserver started at http://127.9.130.1:45525/ using document root <none> and password file <none>
I20260928 02:39:25.512077 18451 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:25.512156 18451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:25.513490 18451 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:25.514101 18484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.514245 18451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:25.514344 18451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "8c811e40cc874533abb5e35157d1d1b2"
format_stamp: "Formatted at 2026-09-28 02:39:24 on dist-test-slave-kkrh"
server_key: "a6a823e558609d347bc547636502dac0"
server_key_iv: "b5877fff319d809be478f5f20f58a74e"
server_key_version: "encryptionkey@0"
I20260928 02:39:25.514691 18451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:25.559739 18451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:25.562206 18451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:25.562348 18451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:25.562598 18451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:25.562896 18451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:25.562929 18451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.562952 18451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:25.562964 18451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.568778 18451 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:32931
I20260928 02:39:25.568825 18597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:32931 every 8 connection(s)
I20260928 02:39:25.569198 18451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:39:25.571128 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 18451
I20260928 02:39:25.571424 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:42589
--local_ip_for_outbound_sockets=127.9.130.2
--tserver_master_addrs=127.9.130.62:34689
--webserver_port=39321
--webserver_interface=127.9.130.2
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.130.2
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:25.577883 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.569934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:43217)
0928 02:39:25.570164 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:25.570199 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:25.570644 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:25.571143 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:25.571147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:25.571875 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:25.572069 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.572074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.573152 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.573153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.573942 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.573945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.574521 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.574522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.574656 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.575227 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:25.575236 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:25.577682 (+ 2446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
W20260928 02:39:25.577939 18601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:25.569978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:43217 (local address 127.9.130.62:34689)
0928 02:39:25.570798 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:25.570804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:25.570830 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:39:25.570920 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:25.570924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:25.570969 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:25.571106 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:25.572192 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.573040 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.574043 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.574435 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.574549 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.577795 (+ 3246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:43217: BlockingRecv error: recv got EOF from 127.9.130.1:43217 (error 108)
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":100,"threads_started":1}
W20260928 02:39:25.578137 18598 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:34689 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.130.62:34689: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:25.582887 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.578387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:53211)
0928 02:39:25.578528 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:25.578538 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:25.578585 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:25.578737 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:25.578737 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:25.578863 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:25.578928 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.578929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.579547 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.579548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.580109 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.580111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.580480 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.580481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.580549 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.581052 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:25.581057 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:25.582797 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
W20260928 02:39:25.582973 18601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:25.578449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:53211 (local address 127.9.130.62:34689)
0928 02:39:25.578507 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:25.578509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:25.578572 (+ 63us) server_negotiation.cc:407] Connection header received
0928 02:39:25.578647 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:25.578648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:25.578661 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:25.578736 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:25.579025 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.579469 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.580193 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.580382 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.580482 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.582899 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:53211: BlockingRecv error: recv got EOF from 127.9.130.1:53211 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:25.588021 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.583294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:41239)
0928 02:39:25.583400 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:25.583410 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:25.583463 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:25.583590 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:25.583590 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:25.583723 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:25.583752 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.583754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.584431 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.584431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.585252 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.585254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.585573 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.585574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.585625 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.586014 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:25.586019 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:25.587938 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260928 02:39:25.588120 18601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:25.583356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:41239 (local address 127.9.130.62:34689)
0928 02:39:25.583403 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:25.583405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:25.583448 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:39:25.583518 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:25.583518 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:25.583526 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:25.583569 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:25.583885 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.584357 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.585330 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.585504 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.585558 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.588033 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:41239: BlockingRecv error: recv got EOF from 127.9.130.1:41239 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 02:39:25.588357 18598 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:39:25.654585 18602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:25.654784 18602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:25.654806 18602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:25.655989 18602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:25.656030 18602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:25.656056 18602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:25.657436 18602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.2.keytab
--principal=oryx/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:42589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=39321
--tserver_master_addrs=127.9.130.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:25.657672 18602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:25.657892 18602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:25.662801 18608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:25.662781 18611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:25.662810 18602 server_base.cc:1034] running on GCE node
W20260928 02:39:25.662777 18609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:25.663203 18602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:25.663471 18602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:25.664625 18602 hybrid_clock.cc:648] HybridClock initialized: now 1790563165664612 us; error 26 us; skew 500 ppm
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563165, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:25.666903 18602 init.cc:377] Logged in from keytab as oryx/127.9.130.2@KRBTEST.COM (short username oryx)
I20260928 02:39:25.667325 18602 webserver.cc:460] Webserver started at http://127.9.130.2:39321/ using document root <none> and password file <none>
I20260928 02:39:25.667542 18602 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:25.667621 18602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:25.669001 18602 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:25.669658 18618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.669855 18602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:25.669968 18602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "d070d5be9c0b46f2ada0d5a6fac1934d"
format_stamp: "Formatted at 2026-09-28 02:39:24 on dist-test-slave-kkrh"
server_key: "37071c5d2dc659033020c01dda898c55"
server_key_iv: "de39a0646ff3dce95c0eb8b4fccd886b"
server_key_version: "encryptionkey@0"
I20260928 02:39:25.670359 18602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:25.696107 18602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:25.698100 18602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:25.698273 18602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:25.698518 18602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:25.698837 18602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:25.698890 18602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.698925 18602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:25.698951 18602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.704939 18602 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:42589
I20260928 02:39:25.704979 18731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:42589 every 8 connection(s)
I20260928 02:39:25.705313 18602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:39:25.706318 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 18602
I20260928 02:39:25.706601 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:35791
--local_ip_for_outbound_sockets=127.9.130.3
--tserver_master_addrs=127.9.130.62:34689
--webserver_port=39989
--webserver_interface=127.9.130.3
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.130.3
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:39:25.713730 18601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:25.706067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:52305 (local address 127.9.130.62:34689)
0928 02:39:25.706129 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:25.706132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:25.706336 (+ 204us) server_negotiation.cc:407] Connection header received
0928 02:39:25.707022 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:25.707023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:25.707042 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:25.707102 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:25.708151 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.708668 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.709778 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.709976 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.710077 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.713683 (+ 3606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:52305: BlockingRecv error: recv got EOF from 127.9.130.2:52305 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:39:25.713820 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.706025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:52305)
0928 02:39:25.706191 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:25.706242 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:25.706901 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:25.707185 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:25.707190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:25.707794 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:25.708034 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.708039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.708791 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.708792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.709660 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.709663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.710088 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.710089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.710230 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.710845 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:25.710853 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:25.713570 (+ 2717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
W20260928 02:39:25.714068 18732 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:34689 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.130.62:34689: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:25.718815 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.714307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:44613)
0928 02:39:25.714444 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:25.714455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:25.714511 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:25.714729 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:25.714730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:25.714865 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:25.714909 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.714910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.715529 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.715530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.716096 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.716098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.716476 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.716477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.716546 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.716981 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:25.716992 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:25.718745 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
W20260928 02:39:25.718905 18601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:25.714353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:44613 (local address 127.9.130.62:34689)
0928 02:39:25.714389 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:25.714391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:25.714507 (+ 116us) server_negotiation.cc:407] Connection header received
0928 02:39:25.714574 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:25.714575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:25.714583 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:25.714722 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:25.714988 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.715439 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.716195 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.716376 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.716478 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.718827 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:44613: BlockingRecv error: recv got EOF from 127.9.130.2:44613 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:25.723605 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.719029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:40211)
0928 02:39:25.719076 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:25.719085 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:25.719116 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:25.719261 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:25.719262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:25.719392 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:25.719424 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.719425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.720068 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.720068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.720863 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.720865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.721230 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.721230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.721319 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.721688 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:25.721693 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:25.723537 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:39:25.723662 18601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:25.719076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:40211 (local address 127.9.130.62:34689)
0928 02:39:25.719102 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:25.719104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:25.719112 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:25.719175 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:25.719175 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:25.719182 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:25.719228 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:25.719486 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.720001 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.720961 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.721152 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.721218 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.723606 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:40211: BlockingRecv error: recv got EOF from 127.9.130.2:40211 (error 108)
Metrics: {"server-negotiator.queue_time_us":12}
W20260928 02:39:25.723704 18732 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:39:25.791047 18735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:25.791249 18735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:25.791277 18735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:25.792493 18735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:25.792534 18735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:25.792562 18735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:25.793968 18735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/krb5kdc/oryx_127.9.130.3.keytab
--principal=oryx/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:35791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=39989
--tserver_master_addrs=127.9.130.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:25.794204 18735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:25.794453 18735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:25.799365 18741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:25.799381 18742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:25.799350 18744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:25.799556 18735 server_base.cc:1034] running on GCE node
I20260928 02:39:25.799914 18735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:25.800163 18735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:25.801309 18735 hybrid_clock.cc:648] HybridClock initialized: now 1790563165801294 us; error 23 us; skew 500 ppm
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563165, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:25.803949 18735 init.cc:377] Logged in from keytab as oryx/127.9.130.3@KRBTEST.COM (short username oryx)
I20260928 02:39:25.804396 18735 webserver.cc:460] Webserver started at http://127.9.130.3:39989/ using document root <none> and password file <none>
I20260928 02:39:25.804684 18735 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:25.804780 18735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:25.806126 18735 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:25.806725 18751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.806896 18735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:25.806995 18735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "2ddfedf97f684c39b20d053e29a57c0b"
format_stamp: "Formatted at 2026-09-28 02:39:25 on dist-test-slave-kkrh"
server_key: "2f5414d4ae219c8f2686118d9258eebd"
server_key_iv: "6d466862dff70e3c2fdc21798bcbe47d"
server_key_version: "encryptionkey@0"
I20260928 02:39:25.807338 18735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:25.847280 18735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:25.849632 18735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:25.849859 18735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:25.850121 18735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:25.850436 18735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:25.850490 18735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.850530 18735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:25.850559 18735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:25.856390 18735 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:35791
I20260928 02:39:25.856431 18864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:35791 every 8 connection(s)
I20260928 02:39:25.856773 18735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:39:25.861874 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 18735
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:39:25.864785 18601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:25.857471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:39975 (local address 127.9.130.62:34689)
0928 02:39:25.857516 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:25.857518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:25.857779 (+ 261us) server_negotiation.cc:407] Connection header received
0928 02:39:25.858330 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:25.858331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:25.858343 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:25.858394 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:25.859371 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.859915 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.860919 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.861111 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.861189 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.864728 (+ 3539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:39975: BlockingRecv error: recv got EOF from 127.9.130.3:39975 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:39:25.864863 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.857475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:39975)
0928 02:39:25.857654 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:25.857691 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:25.858219 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:25.858468 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:25.858472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:25.859040 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:25.859261 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.859266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.860012 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.860013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.860812 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.860814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.861194 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.861195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.861334 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.861995 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:25.862005 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:25.864666 (+ 2661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
W20260928 02:39:25.865070 18865 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:34689 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.130.62:34689: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:39:25.870343 18601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:25.865351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:35537 (local address 127.9.130.62:34689)
0928 02:39:25.865411 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:25.865413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:25.865423 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:39:25.865449 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:25.865450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:25.865459 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:25.865513 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:25.865765 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.866368 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.867040 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.867324 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.867390 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.870256 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:35537: BlockingRecv error: recv got EOF from 127.9.130.3:35537 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:39:25.870529 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.865282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:35537)
0928 02:39:25.865332 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:25.865341 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:25.865403 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:25.865517 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:25.865518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:25.865639 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:25.865684 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.865685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.866460 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.866461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.866957 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.866958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.867403 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.867405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.867495 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.868102 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:25.868107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:25.870446 (+ 2339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 02:39:25.871474 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.862412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.9.130.62:34689)
0928 02:39:25.862528 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:25.862530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:25.862622 (+ 92us) server_negotiation.cc:407] Connection header received
0928 02:39:25.862680 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:25.862681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:25.862694 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:25.862768 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:25.863521 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.864272 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.865372 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.865599 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.865696 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.866954 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:25.866958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:25.866959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:25.866970 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:25.869142 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:25.869604 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:25.869608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:25.869608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:25.869745 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:25.870055 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:25.870056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:25.870057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:25.870650 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:25.870793 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:25.871259 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 02:39:25.871376 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:25 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:25.877125 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:25.870787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:43703)
0928 02:39:25.870920 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:25.870931 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:25.870996 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:25.871191 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:25.871192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:25.871341 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:25.871380 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.871382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.872473 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.872475 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.873550 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:25.873553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.874010 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.874010 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:25.874080 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.874706 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:25.874711 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:25.877043 (+ 2332us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
W20260928 02:39:25.877249 18865 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:39:25.877283 18601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:25.870844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:43703 (local address 127.9.130.62:34689)
0928 02:39:25.870885 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:25.870887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:25.870970 (+ 83us) server_negotiation.cc:407] Connection header received
0928 02:39:25.871096 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:25.871096 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:25.871107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:25.871178 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:25.871449 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.872387 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.873629 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:25.873931 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:25.874011 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:25.877125 (+ 3114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:43703: BlockingRecv error: recv got EOF from 127.9.130.3:43703 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 02:39:26 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:26 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:26.593605 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:26.588774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:46191)
0928 02:39:26.588906 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:26.588916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:26.588968 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:26.589215 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:26.589216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:26.589383 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:26.589430 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:26.589432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:26.590194 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:26.590195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:26.590754 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:26.590756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:26.591146 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:26.591148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:26.591216 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:26.591807 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:26.591850 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:26.593539 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
W20260928 02:39:26.593698 18872 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:26.588889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:46191 (local address 127.9.130.62:34689)
0928 02:39:26.589027 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:26.589030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:26.589040 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:39:26.589109 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:26.589109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:26.589123 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:26.589203 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:26.589499 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:26.589991 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:26.590839 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:26.591058 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:26.591129 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:26.593627 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:46191: BlockingRecv error: recv got EOF from 127.9.130.1:46191 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":107,"threads_started":1}
Sep 28 02:39:26 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:26 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:26.728619 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:26.724049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:36595)
0928 02:39:26.724228 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:26.724238 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:26.724293 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:26.724438 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:26.724439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:26.724596 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:26.724642 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:26.724644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:26.725317 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:26.725318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:26.725893 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:26.725895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:26.726268 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:26.726270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:26.726320 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:26.726782 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:26.726800 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:26.728553 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
W20260928 02:39:26.728796 18872 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:26.724130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:36595 (local address 127.9.130.62:34689)
0928 02:39:26.724197 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:26.724199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:26.724295 (+ 96us) server_negotiation.cc:407] Connection header received
0928 02:39:26.724347 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:26.724347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:26.724357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:26.724426 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:26.724750 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:26.725247 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:26.725973 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:26.726173 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:26.726231 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:26.728679 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:36595: BlockingRecv error: recv got EOF from 127.9.130.2:36595 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 02:39:26 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:26 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:26.882512 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:26.877619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:40249)
0928 02:39:26.877796 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:26.877808 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:26.877909 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:26.878093 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:26.878094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:26.878291 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:26.878338 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:26.878339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:26.879011 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:26.879013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:26.879563 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:26.879564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:26.879985 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:26.879986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:26.880050 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:26.880555 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:26.880560 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:26.882393 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":91,"threads_started":1}
W20260928 02:39:26.882527 18872 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:26.877716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:40249 (local address 127.9.130.62:34689)
0928 02:39:26.877792 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:26.877795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:26.877891 (+ 96us) server_negotiation.cc:407] Connection header received
0928 02:39:26.877966 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:26.877967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:26.877979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:26.878077 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:26.878431 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:26.878926 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:26.879650 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:26.879901 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:26.879984 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:26.882468 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:40249: BlockingRecv error: recv got EOF from 127.9.130.3:40249 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:39:27 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:27 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:27.598918 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:27.594209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:34799)
0928 02:39:27.594357 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:27.594369 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:27.594443 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:27.594623 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:27.594623 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:27.594883 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:27.594938 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:27.594940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:27.595611 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:27.595612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:27.596231 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:27.596233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:27.596609 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:27.596610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:27.596668 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:27.597123 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:27.597127 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:27.598824 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
W20260928 02:39:27.599009 18876 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:27.594282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:34799 (local address 127.9.130.62:34689)
0928 02:39:27.594398 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:27.594400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:27.594413 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:39:27.594506 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:27.594507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:27.594521 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:27.594616 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:27.595009 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:27.595521 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:27.596318 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:27.596522 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:27.596608 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:27.598913 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:34799: BlockingRecv error: recv got EOF from 127.9.130.1:34799 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":52,"threads_started":1}
Sep 28 02:39:27 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:27 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:27.734140 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:27.729167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:33325)
0928 02:39:27.729320 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:27.729330 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:27.729384 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:27.729510 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:27.729511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:27.729667 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:27.729713 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:27.729715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:27.730386 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:27.730387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:27.731290 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:27.731293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:27.731699 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:27.731700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:27.731747 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:27.732364 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:27.732369 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:27.734083 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
W20260928 02:39:27.734364 18876 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:27.729233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:33325 (local address 127.9.130.62:34689)
0928 02:39:27.729306 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:27.729308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:27.729389 (+ 81us) server_negotiation.cc:407] Connection header received
0928 02:39:27.729430 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:27.729431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:27.729444 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:27.729503 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:27.729772 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:27.730302 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:27.731376 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:27.731598 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:27.731664 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:27.734290 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:33325: BlockingRecv error: recv got EOF from 127.9.130.2:33325 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 02:39:27 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:27 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:27.887437 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:27.883052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:47429)
0928 02:39:27.883234 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:27.883248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:27.883314 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:27.883449 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:27.883450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:27.883616 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:27.883663 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:27.883664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:27.884323 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:27.884324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:27.884850 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:27.884852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:27.885207 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:27.885208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:27.885245 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:27.885706 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:27.885710 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:27.887377 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
W20260928 02:39:27.887533 18876 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:27.883206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:47429 (local address 127.9.130.62:34689)
0928 02:39:27.883270 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:27.883272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:27.883299 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:39:27.883369 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:27.883370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:27.883381 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:27.883430 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:27.883724 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:27.884258 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:27.884960 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:27.885140 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:27.885195 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:27.887468 (+ 2273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:47429: BlockingRecv error: recv got EOF from 127.9.130.3:47429 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 02:39:28 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:28 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:28.604079 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:28.599414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:52879)
0928 02:39:28.599573 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:28.599583 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:28.599636 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:28.599793 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:28.599794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:28.600000 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:28.600049 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:28.600051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:28.600714 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:28.600714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:28.601293 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:28.601296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:28.601667 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:28.601668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:28.601709 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:28.602171 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:28.602175 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:28.604013 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
W20260928 02:39:28.604132 18880 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:28.599511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:52879 (local address 127.9.130.62:34689)
0928 02:39:28.599639 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:28.599641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:28.599674 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:39:28.599701 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:28.599701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:28.599711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:28.599765 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:28.600131 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:28.600644 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:28.601375 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:28.601580 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:28.601660 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:28.604079 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:52879: BlockingRecv error: recv got EOF from 127.9.130.1:52879 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
Sep 28 02:39:28 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:28 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:28.739565 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:28.734622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:37721)
0928 02:39:28.734748 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:28.734759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:28.734811 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:28.734990 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:28.734990 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:28.735150 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:28.735196 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:28.735197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:28.735924 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:28.735925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:28.736752 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:28.736769 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:28.737155 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:28.737156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:28.737197 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:28.737643 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:28.737648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:28.739485 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
W20260928 02:39:28.739732 18880 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:28.734727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:37721 (local address 127.9.130.62:34689)
0928 02:39:28.734782 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:28.734784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:28.734818 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:39:28.734870 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:28.734871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:28.734883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:28.734973 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:28.735270 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:28.735769 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:28.736868 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:28.737086 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:28.737144 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:28.739617 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:37721: BlockingRecv error: recv got EOF from 127.9.130.2:37721 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 02:39:28 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:28 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:28.892805 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:28.888006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:38503)
0928 02:39:28.888203 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:28.888218 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:28.888300 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:28.888505 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:28.888506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:28.888708 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:28.888754 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:28.888756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:28.889479 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:28.889480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:28.890010 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:28.890012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:28.890387 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:28.890388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:28.890428 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:28.890852 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:28.890855 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:28.892700 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
W20260928 02:39:28.892836 18880 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:28.888142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:38503 (local address 127.9.130.62:34689)
0928 02:39:28.888183 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:28.888185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:28.888358 (+ 173us) server_negotiation.cc:407] Connection header received
0928 02:39:28.888380 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:28.888381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:28.888393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:28.888482 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:28.888818 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:28.889411 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:28.890081 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:28.890313 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:28.890379 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:28.892773 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:38503: BlockingRecv error: recv got EOF from 127.9.130.3:38503 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 28 02:39:29 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:29 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:29.608731 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:29.604515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:52665)
0928 02:39:29.604663 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:29.604674 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:29.604733 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:29.604856 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:29.604856 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:29.605005 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:29.605049 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:29.605050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:29.605684 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:29.605684 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:29.606183 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:29.606185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:29.606514 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:29.606515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:29.606554 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:29.606987 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:29.606991 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:29.608674 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
W20260928 02:39:29.608882 18884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:29.604579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:52665 (local address 127.9.130.62:34689)
0928 02:39:29.604689 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:29.604691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:29.604718 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:39:29.604782 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:29.604783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:29.604793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:29.604842 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:29.605125 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:29.605619 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:29.606254 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:29.606447 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:29.606502 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:29.608787 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:52665: BlockingRecv error: recv got EOF from 127.9.130.1:52665 (error 108)
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":46,"threads_started":1}
Sep 28 02:39:29 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:29 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:29.748188 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:29.740081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:54179)
0928 02:39:29.740235 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:29.740246 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:29.740311 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:29.740439 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:29.740440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:29.740596 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:29.740643 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:29.740645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:29.741293 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:29.741294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:29.741828 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:29.741830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:29.742145 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:29.742145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:29.742183 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:29.746272 (+ 4089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:29.746276 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:29.748098 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
W20260928 02:39:29.748286 18884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:29.740150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:54179 (local address 127.9.130.62:34689)
0928 02:39:29.740240 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:29.740242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:29.740310 (+ 68us) server_negotiation.cc:407] Connection header received
0928 02:39:29.740366 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:29.740366 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:29.740377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:29.740416 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:29.740704 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:29.741211 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:29.741900 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:29.742084 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:29.742133 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:29.748222 (+ 6089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:54179: BlockingRecv error: recv got EOF from 127.9.130.2:54179 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 02:39:29 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:29 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:29.898126 18886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:29.893323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:45623)
0928 02:39:29.893512 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:29.893523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:29.893615 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:29.893782 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:29.893783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:29.893957 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:29.894003 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:29.894004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:29.894838 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:29.894839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:29.895386 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:29.895388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:29.895718 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:29.895719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:29.895762 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:29.896239 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:29.896244 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:29.898072 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":115,"threads_started":1}
W20260928 02:39:29.898327 18884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:29.893413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:45623 (local address 127.9.130.62:34689)
0928 02:39:29.893493 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:29.893495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:29.893593 (+ 98us) server_negotiation.cc:407] Connection header received
0928 02:39:29.893680 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:29.893681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:29.893694 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:29.893763 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:29.894109 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:29.894753 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:29.895461 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:29.895638 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:29.895693 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:29.898216 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:45623: BlockingRecv error: recv got EOF from 127.9.130.3:45623 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 02:39:30 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:30 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:30.613828 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:30.609138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:45837)
0928 02:39:30.609276 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:30.609289 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:30.609374 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:30.609543 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:30.609543 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:30.609698 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:30.609745 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:30.609746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:30.610504 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:30.610506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:30.611025 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:30.611027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:30.611468 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:30.611470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:30.611519 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:30.612039 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:30.612044 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:30.613768 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":51,"threads_started":1}
W20260928 02:39:30.613906 18888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:30.609243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:45837 (local address 127.9.130.62:34689)
0928 02:39:30.609373 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:30.609375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:30.609383 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:30.609422 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:30.609422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:30.609432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:30.609523 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:30.609818 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:30.610436 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:30.611097 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:30.611377 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:30.611445 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:30.613855 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:45837: BlockingRecv error: recv got EOF from 127.9.130.1:45837 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
Sep 28 02:39:30 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:30 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:30.753340 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:30.748707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:36625)
0928 02:39:30.748882 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:30.748893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:30.748959 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:30.749092 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:30.749093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:30.749254 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:30.749300 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:30.749302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:30.749956 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:30.749957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:30.750483 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:30.750484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:30.750850 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:30.750851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:30.750902 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:30.751334 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:30.751340 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:30.753279 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
W20260928 02:39:30.753407 18888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:30.748837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:36625 (local address 127.9.130.62:34689)
0928 02:39:30.748882 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:30.748885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:30.748937 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:39:30.749015 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:30.749016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:30.749026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:30.749068 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:30.749391 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:30.749883 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:30.750559 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:30.750746 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:30.750816 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:30.753359 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:36625: BlockingRecv error: recv got EOF from 127.9.130.2:36625 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 02:39:30 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:30 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:30.903517 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:30.898740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:36677)
0928 02:39:30.898896 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:30.898910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:30.898979 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:30.899167 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:30.899168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:30.899438 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:30.899496 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:30.899499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:30.900361 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:30.900362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:30.900873 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:30.900875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:30.901228 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:30.901229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:30.901266 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:30.901748 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:30.901752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:30.903461 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
W20260928 02:39:30.903676 18888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:30.898910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:36677 (local address 127.9.130.62:34689)
0928 02:39:30.898975 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:30.898995 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:30.899010 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:39:30.899040 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:30.899041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:30.899059 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:30.899140 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:30.899585 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:30.900260 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:30.900975 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:30.901157 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:30.901219 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:30.903576 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:36677: BlockingRecv error: recv got EOF from 127.9.130.3:36677 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 02:39:31 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:31 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:31.618739 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:31.614283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:60213)
0928 02:39:31.614402 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:31.614413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:31.614489 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:31.614679 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:31.614679 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:31.614830 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:31.614880 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:31.614881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:31.615508 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:31.615509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:31.616052 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:31.616054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:31.616371 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:31.616371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:31.616410 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:31.616825 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:31.616829 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:31.618645 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
W20260928 02:39:31.618783 18892 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:31.614330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:60213 (local address 127.9.130.62:34689)
0928 02:39:31.614453 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:31.614455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:31.614464 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:39:31.614580 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:31.614581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:31.614593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:31.614662 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:31.614945 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:31.615429 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:31.616115 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:31.616302 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:31.616352 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:31.618714 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:60213: BlockingRecv error: recv got EOF from 127.9.130.1:60213 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
Sep 28 02:39:31 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:31 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:31.758653 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:31.753954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:43317)
0928 02:39:31.754112 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:31.754127 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:31.754205 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:31.754325 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:31.754326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:31.754485 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:31.754551 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:31.754552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:31.755377 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:31.755378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:31.755997 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:31.756000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:31.756376 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:31.756378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:31.756435 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:31.756886 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:31.756890 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:31.758594 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
W20260928 02:39:31.758725 18892 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:31.754086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:43317 (local address 127.9.130.62:34689)
0928 02:39:31.754155 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:31.754157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:31.754188 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:39:31.754255 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:31.754256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:31.754266 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:31.754315 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:31.754629 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:31.755270 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:31.756078 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:31.756260 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:31.756316 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:31.758679 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:43317: BlockingRecv error: recv got EOF from 127.9.130.2:43317 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 02:39:31 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:31 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:31.908747 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:31.904172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:34631)
0928 02:39:31.904298 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:31.904329 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:31.904382 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:31.904582 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:31.904583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:31.904817 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:31.904881 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:31.904883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:31.905544 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:31.905545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:31.906059 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:31.906061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:31.906364 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:31.906365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:31.906402 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:31.906912 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:31.906917 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:31.908685 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
W20260928 02:39:31.908916 18892 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:31.904236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:34631 (local address 127.9.130.62:34689)
0928 02:39:31.904310 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:31.904312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:31.904391 (+ 79us) server_negotiation.cc:407] Connection header received
0928 02:39:31.904462 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:31.904463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:31.904490 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:31.904562 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:31.904969 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:31.905481 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:31.906131 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:31.906305 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:31.906355 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:31.908806 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:34631: BlockingRecv error: recv got EOF from 127.9.130.3:34631 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 02:39:32 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:32 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:32.623852 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:32.619243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:34409)
0928 02:39:32.619383 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:32.619396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:32.619457 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:32.619608 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:32.619608 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:32.619760 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:32.619808 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:32.619830 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:32.620458 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:32.620458 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:32.620968 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:32.620969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:32.621416 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:32.621416 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:32.621481 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:32.621885 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:32.621890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:32.623728 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
W20260928 02:39:32.623905 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:32.619300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:34409 (local address 127.9.130.62:34689)
0928 02:39:32.619434 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:32.619435 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:32.619444 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:39:32.619502 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:32.619502 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:32.619512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:32.619600 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:32.619893 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:32.620391 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:32.621079 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:32.621330 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:32.621413 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:32.623811 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:34409: BlockingRecv error: recv got EOF from 127.9.130.1:34409 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
Sep 28 02:39:32 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:32 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:32.764098 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:32.759240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:36937)
0928 02:39:32.759399 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:32.759451 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:32.759519 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:32.759664 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:32.759665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:32.759898 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:32.759951 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:32.759953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:32.760654 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:32.760655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:32.761189 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:32.761191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:32.761524 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:32.761525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:32.761576 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:32.762265 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:32.762269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:32.764039 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
W20260928 02:39:32.764225 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:32.759337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:36937 (local address 127.9.130.62:34689)
0928 02:39:32.759422 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:32.759424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:32.759520 (+ 96us) server_negotiation.cc:407] Connection header received
0928 02:39:32.759571 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:32.759572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:32.759584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:32.759641 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:32.760024 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:32.760526 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:32.761268 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:32.761453 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:32.761506 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:32.764142 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:36937: BlockingRecv error: recv got EOF from 127.9.130.2:36937 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 02:39:32 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:32 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:32.914342 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:32.909243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:41637)
0928 02:39:32.909432 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:32.909442 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:32.909507 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:32.909710 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:32.909711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:32.909863 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:32.909911 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:32.909912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:32.910709 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:32.910710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:32.911241 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:32.911242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:32.911612 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:32.911613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:32.911658 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:32.912151 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:32.912156 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:32.914275 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
W20260928 02:39:32.914461 18896 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:32.909333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:41637 (local address 127.9.130.62:34689)
0928 02:39:32.909384 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:32.909386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:32.909514 (+ 128us) server_negotiation.cc:407] Connection header received
0928 02:39:32.909585 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:32.909586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:32.909598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:32.909700 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:32.909985 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:32.910620 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:32.911336 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:32.911535 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:32.911599 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:32.914363 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:41637: BlockingRecv error: recv got EOF from 127.9.130.3:41637 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 02:39:33 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:33 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:33.629026 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:33.624299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:44473)
0928 02:39:33.624438 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:33.624448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:33.624498 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:33.624731 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:33.624731 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:33.624885 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:33.624932 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:33.624933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:33.625593 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:33.625594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:33.626119 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:33.626120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:33.626454 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:33.626455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:33.626493 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:33.626923 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:33.626928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:33.628970 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
W20260928 02:39:33.629140 18900 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:33.624407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:44473 (local address 127.9.130.62:34689)
0928 02:39:33.624608 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:33.624610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:33.624618 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:33.624639 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:33.624639 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:33.624650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:33.624714 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:33.625021 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:33.625527 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:33.626195 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:33.626388 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:33.626445 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:33.629058 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:44473: BlockingRecv error: recv got EOF from 127.9.130.1:44473 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":152,"threads_started":1}
Sep 28 02:39:33 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:33 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:33.769424 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:33.764610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:34449)
0928 02:39:33.764770 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:33.764782 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:33.764847 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:33.764980 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:33.764981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:33.765142 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:33.765190 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:33.765191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:33.765960 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:33.765961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:33.766489 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:33.766490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:33.766820 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:33.766821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:33.766860 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:33.767281 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:33.767285 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:33.769329 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
W20260928 02:39:33.769460 18900 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:33.764695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:34449 (local address 127.9.130.62:34689)
0928 02:39:33.764764 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:33.764766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:33.764833 (+ 67us) server_negotiation.cc:407] Connection header received
0928 02:39:33.764905 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:33.764906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:33.764916 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:33.764958 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:33.765253 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:33.765890 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:33.766559 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:33.766757 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:33.766806 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:33.769404 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:34449: BlockingRecv error: recv got EOF from 127.9.130.2:34449 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 02:39:33 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:33 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:33.919312 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:33.914936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:59725)
0928 02:39:33.915064 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:33.915083 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:33.915139 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:33.915280 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:33.915281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:33.915430 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:33.915479 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:33.915480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:33.916107 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:33.916107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:33.916606 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:33.916607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:33.916950 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:33.916951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:33.916991 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:33.917510 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:33.917515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:33.919256 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
W20260928 02:39:33.919451 18900 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:33.914997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:59725 (local address 127.9.130.62:34689)
0928 02:39:33.915086 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:33.915088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:33.915148 (+ 60us) server_negotiation.cc:407] Connection header received
0928 02:39:33.915185 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:33.915186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:33.915197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:33.915255 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:33.915540 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:33.916042 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:33.916698 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:33.916889 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:33.916942 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:33.919371 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:59725: BlockingRecv error: recv got EOF from 127.9.130.3:59725 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:39:34 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:34 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:34.634393 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:34.629577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:54437)
0928 02:39:34.629720 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:34.629732 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:34.629801 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:34.629963 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:34.629964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:34.630173 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:34.630219 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:34.630220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:34.630879 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:34.630880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:34.631408 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:34.631410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:34.631871 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:34.631872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:34.631928 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:34.632396 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:34.632401 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:34.634317 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
W20260928 02:39:34.634514 18904 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:34.629615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:54437 (local address 127.9.130.62:34689)
0928 02:39:34.629719 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:34.629721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:34.629799 (+ 78us) server_negotiation.cc:407] Connection header received
0928 02:39:34.629852 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:34.629853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:34.629866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:34.629940 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:34.630290 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:34.630800 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:34.631508 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:34.631739 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:34.631808 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:34.634433 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:54437: BlockingRecv error: recv got EOF from 127.9.130.1:54437 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":43,"threads_started":1}
Sep 28 02:39:34 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:34 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:34.778184 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:34.770076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:47699)
0928 02:39:34.770247 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:34.770261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:34.770347 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:34.770515 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:34.770516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:34.770685 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:34.770734 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:34.770735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:34.771407 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:34.771407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:34.771937 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:34.771938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:34.772281 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:34.772282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:34.772319 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:34.776355 (+ 4036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:34.776360 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:34.778128 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
W20260928 02:39:34.778353 18904 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:34.770132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:47699 (local address 127.9.130.62:34689)
0928 02:39:34.770251 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:34.770254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:34.770341 (+ 87us) server_negotiation.cc:407] Connection header received
0928 02:39:34.770412 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:34.770413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:34.770430 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:34.770501 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:34.770835 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:34.771334 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:34.772014 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:34.772213 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:34.772273 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:34.778250 (+ 5977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:47699: BlockingRecv error: recv got EOF from 127.9.130.2:47699 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 28 02:39:34 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:34 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:34.924768 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:34.919796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:51429)
0928 02:39:34.919988 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:34.920007 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:34.920073 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:34.920198 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:34.920199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:34.920358 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:34.920407 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:34.920409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:34.921035 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:34.921035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:34.921591 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:34.921594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:34.922117 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:34.922118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:34.922189 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:34.922714 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:34.922720 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:34.924672 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":116,"threads_started":1}
W20260928 02:39:34.924815 18904 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:34.919944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:51429 (local address 127.9.130.62:34689)
0928 02:39:34.920008 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:34.920010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:34.920069 (+ 59us) server_negotiation.cc:407] Connection header received
0928 02:39:34.920119 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:34.920120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:34.920131 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:34.920180 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:34.920466 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:34.920965 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:34.921701 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:34.922039 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:34.922107 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:34.924750 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:51429: BlockingRecv error: recv got EOF from 127.9.130.3:51429 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 02:39:35 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:35 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:35.639933 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:35.634914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:58313)
0928 02:39:35.635100 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:35.635114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:35.635180 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:35.635310 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:35.635311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:35.635477 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:35.635527 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:35.635528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:35.636313 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:35.636314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:35.636878 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:35.636881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:35.637246 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:35.637247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:35.637296 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:35.637730 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:35.637735 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:35.639806 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":94,"threads_started":1}
W20260928 02:39:35.640034 18908 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:35.634986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:58313 (local address 127.9.130.62:34689)
0928 02:39:35.635140 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:35.635142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:35.635154 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:39:35.635224 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:35.635225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:35.635235 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:35.635295 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:35.635591 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:35.636250 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:35.636972 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:35.637158 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:35.637213 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:35.639942 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:58313: BlockingRecv error: recv got EOF from 127.9.130.1:58313 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
Sep 28 02:39:35 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:35 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:35.786092 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:35.778643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:51177)
0928 02:39:35.778816 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:35.778828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:35.778914 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:35.779043 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:35.779044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:35.779223 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:35.779267 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:35.779268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:35.780022 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:35.780023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:35.780551 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:35.780553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:35.780933 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:35.780934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:35.781003 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:35.784267 (+ 3264us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:35.784272 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:35.785984 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
W20260928 02:39:35.786167 18908 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:35.778729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:51177 (local address 127.9.130.62:34689)
0928 02:39:35.778804 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:35.778806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:35.778900 (+ 94us) server_negotiation.cc:407] Connection header received
0928 02:39:35.778972 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:35.778973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:35.778986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:35.779038 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:35.779358 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:35.779923 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:35.780638 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:35.780841 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:35.780934 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:35.786105 (+ 5171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:51177: BlockingRecv error: recv got EOF from 127.9.130.2:51177 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 02:39:35 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:35 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:35.932099 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:35.925485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:39801)
0928 02:39:35.925615 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:35.925625 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:35.925683 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:35.925924 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:35.925925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:35.926187 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:35.926263 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:35.926265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:35.927405 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:35.927419 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:35.928396 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:35.928399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:35.928882 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:35.928884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:35.928939 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:35.929551 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:35.929556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:35.932032 (+ 2476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
W20260928 02:39:35.932181 18908 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:35.925596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:39801 (local address 127.9.130.62:34689)
0928 02:39:35.925651 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:35.925654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:35.925668 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:39:35.925758 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:35.925759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:35.925771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:35.925914 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:35.926500 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:35.927327 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:35.928494 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:35.928801 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:35.928882 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:35.932119 (+ 3237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:39801: BlockingRecv error: recv got EOF from 127.9.130.3:39801 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 02:39:36 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:36 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:36.645213 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:36.640505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:37499)
0928 02:39:36.640661 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:36.640671 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:36.640721 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:36.640845 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:36.640846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:36.641013 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:36.641056 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:36.641057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:36.641741 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:36.641741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:36.642271 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:36.642272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:36.642611 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:36.642612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:36.642662 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:36.643120 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:36.643124 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:36.645146 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
W20260928 02:39:36.645251 18912 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:36.640567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:37499 (local address 127.9.130.62:34689)
0928 02:39:36.640716 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:36.640718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:36.640727 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:39:36.640773 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:36.640774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:36.640786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:36.640835 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:36.641163 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:36.641670 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:36.642344 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:36.642527 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:36.642598 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:36.645199 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:37499: BlockingRecv error: recv got EOF from 127.9.130.1:37499 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
Sep 28 02:39:36 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:36 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:36.791270 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:36.786552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:38255)
0928 02:39:36.786736 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:36.786748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:36.786839 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:36.787000 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:36.787001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:36.787189 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:36.787232 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:36.787234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:36.787919 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:36.787920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:36.788455 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:36.788457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:36.788863 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:36.788864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:36.788928 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:36.789429 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:36.789434 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:36.791196 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":110,"threads_started":1}
W20260928 02:39:36.791390 18912 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:36.786641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:38255 (local address 127.9.130.62:34689)
0928 02:39:36.786728 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:36.786730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:36.786821 (+ 91us) server_negotiation.cc:407] Connection header received
0928 02:39:36.786925 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:36.786926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:36.786938 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:36.786980 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:36.787314 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:36.787812 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:36.788571 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:36.788779 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:36.788863 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:36.791293 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:38255: BlockingRecv error: recv got EOF from 127.9.130.2:38255 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 02:39:36 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:36 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:36.937300 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:36.932680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:52245)
0928 02:39:36.932826 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:36.932840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:36.932892 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:36.933054 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:36.933054 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:36.933213 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:36.933256 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:36.933258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:36.933877 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:36.933877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:36.934435 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:36.934437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:36.934762 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:36.934763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:36.934807 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:36.935278 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:36.935283 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:36.937182 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
W20260928 02:39:36.937363 18912 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:36.932737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:52245 (local address 127.9.130.62:34689)
0928 02:39:36.932809 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:36.932811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:36.932908 (+ 97us) server_negotiation.cc:407] Connection header received
0928 02:39:36.932930 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:36.932931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:36.932941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:36.932993 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:36.933319 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:36.933805 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:36.934512 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:36.934695 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:36.934748 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:36.937291 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:52245: BlockingRecv error: recv got EOF from 127.9.130.3:52245 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 02:39:37 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:37 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:37.650055 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:37.645678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:43405)
0928 02:39:37.645806 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:37.645825 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:37.645872 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:37.646056 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:37.646058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:37.646228 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:37.646273 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:37.646274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:37.646906 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:37.646907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:37.647446 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:37.647448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:37.647799 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:37.647800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:37.647897 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:37.648348 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:37.648352 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:37.649999 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
W20260928 02:39:37.650204 18916 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:37.645792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:43405 (local address 127.9.130.62:34689)
0928 02:39:37.645915 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:37.645917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:37.645925 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:37.645958 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:37.645958 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:37.645972 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:37.646037 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:37.646341 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:37.646841 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:37.647524 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:37.647730 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:37.647786 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:37.650113 (+ 2327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:43405: BlockingRecv error: recv got EOF from 127.9.130.1:43405 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
Sep 28 02:39:37 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:37 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:37.797343 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:37.791686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:41001)
0928 02:39:37.791898 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:37.791911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:37.792011 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:37.792187 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:37.792188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:37.792388 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:37.792433 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:37.792435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:37.793281 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:37.793282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:37.794030 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:37.794032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:37.794456 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:37.794457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:37.794530 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:37.795092 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:37.795097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:37.797239 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
W20260928 02:39:37.797394 18916 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:37.791782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:41001 (local address 127.9.130.62:34689)
0928 02:39:37.791879 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:37.791881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:37.791992 (+ 111us) server_negotiation.cc:407] Connection header received
0928 02:39:37.792088 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:37.792089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:37.792108 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:37.792189 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:37.792521 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:37.793135 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:37.794122 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:37.794346 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:37.794444 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:37.797325 (+ 2881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:41001: BlockingRecv error: recv got EOF from 127.9.130.2:41001 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:39:37 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:37 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:37.942117 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:37.937775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:40771)
0928 02:39:37.937911 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:37.937929 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:37.937987 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:37.938107 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:37.938108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:37.938274 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:37.938318 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:37.938320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:37.938941 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:37.938942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:37.939501 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:37.939503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:37.939891 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:37.939892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:37.939941 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:37.940334 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:37.940338 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:37.942054 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
W20260928 02:39:37.942243 18916 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:37.937829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:40771 (local address 127.9.130.62:34689)
0928 02:39:37.937901 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:37.937903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:37.937985 (+ 82us) server_negotiation.cc:407] Connection header received
0928 02:39:37.938037 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:37.938038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:37.938049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:37.938089 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:37.938382 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:37.938872 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:37.939579 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:37.939771 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:37.939874 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:37.942151 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:40771: BlockingRecv error: recv got EOF from 127.9.130.3:40771 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 02:39:38 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:38 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:38.655529 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:38.650530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:40987)
0928 02:39:38.650661 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:38.650673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:38.650733 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:38.650904 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:38.650905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:38.651065 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:38.651109 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:38.651110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:38.651881 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:38.651882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:38.652601 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:38.652604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:38.653079 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:38.653079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:38.653122 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:38.653584 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:38.653589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:38.655462 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
W20260928 02:39:38.655699 18920 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:38.650644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:40987 (local address 127.9.130.62:34689)
0928 02:39:38.650769 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:38.650771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:38.650786 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:39:38.650810 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:38.650810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:38.650820 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:38.650886 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:38.651204 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:38.651753 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:38.652721 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:38.653008 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:38.653067 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:38.655588 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:40987: BlockingRecv error: recv got EOF from 127.9.130.1:40987 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
Sep 28 02:39:38 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:38 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:38.802372 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:38.797880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:53919)
0928 02:39:38.798023 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:38.798031 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:38.798080 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:38.798205 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:38.798205 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:38.798366 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:38.798410 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:38.798411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:38.799037 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:38.799038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:38.799591 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:38.799593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:38.800035 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:38.800036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:38.800085 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:38.800544 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:38.800548 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:38.802296 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":56,"threads_started":1}
W20260928 02:39:38.802522 18920 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:38.797986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:53919 (local address 127.9.130.62:34689)
0928 02:39:38.798036 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:38.798043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:38.798068 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:39:38.798133 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:38.798134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:38.798143 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:38.798192 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:38.798473 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:38.798962 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:38.799688 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:38.799949 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:38.800013 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:38.802447 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:53919: BlockingRecv error: recv got EOF from 127.9.130.2:53919 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 02:39:38 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:38 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:38.947548 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:38.942659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:55677)
0928 02:39:38.942840 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:38.942856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:38.942922 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:38.943084 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:38.943085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:38.943268 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:38.943314 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:38.943316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:38.944043 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:38.944044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:38.944663 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:38.944665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:38.945003 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:38.945004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:38.945046 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:38.945533 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:38.945538 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:38.947474 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":122,"threads_started":1}
W20260928 02:39:38.947659 18920 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:38.942797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:55677 (local address 127.9.130.62:34689)
0928 02:39:38.942863 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:38.942865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:38.942907 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:39:38.942992 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:38.942993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:38.943004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:38.943063 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:38.943414 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:38.943946 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:38.944745 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:38.944933 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:38.944990 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:38.947566 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:55677: BlockingRecv error: recv got EOF from 127.9.130.3:55677 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 02:39:39 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:39 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:39.660794 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:39.656211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:52193)
0928 02:39:39.656357 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:39.656367 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:39.656436 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:39.656553 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:39.656554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:39.656716 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:39.656762 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:39.656763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:39.657421 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:39.657422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:39.657947 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:39.657948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:39.658297 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:39.658298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:39.658336 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:39.658775 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:39.658779 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:39.660687 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
W20260928 02:39:39.660848 18924 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:39.656255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:52193 (local address 127.9.130.62:34689)
0928 02:39:39.656366 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:39.656368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:39.656438 (+ 70us) server_negotiation.cc:407] Connection header received
0928 02:39:39.656480 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:39.656481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:39.656491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:39.656550 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:39.656823 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:39.657342 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:39.658033 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:39.658228 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:39.658286 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:39.660775 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:52193: BlockingRecv error: recv got EOF from 127.9.130.1:52193 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
Sep 28 02:39:39 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:39 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:39.810118 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:39.802818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:56127)
0928 02:39:39.802942 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:39.802956 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:39.803008 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:39.803127 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:39.803127 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:39.803288 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:39.803332 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:39.803333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:39.803966 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:39.803967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:39.804489 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:39.804490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:39.804804 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:39.804804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:39.804843 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:39.808110 (+ 3267us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:39.808121 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:39.810048 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
W20260928 02:39:39.810232 18924 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:39.802881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:56127 (local address 127.9.130.62:34689)
0928 02:39:39.802979 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:39.802985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:39.802995 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:39:39.803064 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:39.803065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:39.803075 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:39.803123 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:39.803387 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:39.803900 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:39.804559 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:39.804741 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:39.804793 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:39.810152 (+ 5359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:56127: BlockingRecv error: recv got EOF from 127.9.130.2:56127 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:39:39 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:39 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.3@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:39.953346 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:39.948013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.3:60619)
0928 02:39:39.948311 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:39.948323 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:39.948448 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:39.948602 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:39.948603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:39.948787 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:39.948831 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:39.948833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:39.949498 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:39.949499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:39.950036 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:39.950038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:39.950427 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:39.950428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:39.950495 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:39.951018 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:39.951023 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:39.953250 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":105,"threads_started":1}
W20260928 02:39:39.953385 18924 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:39.948113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:60619 (local address 127.9.130.62:34689)
0928 02:39:39.948299 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:39.948302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:39.948434 (+ 132us) server_negotiation.cc:407] Connection header received
0928 02:39:39.948506 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:39.948507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:39.948521 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:39.948595 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:39.948926 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:39.949415 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:39.950131 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:39.950322 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:39.950429 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:39.953326 (+ 2897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.3:60619: BlockingRecv error: recv got EOF from 127.9.130.3:60619 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Sep 28 02:39:40 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:40 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.1@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:40.665860 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:40.661320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.1:55821)
0928 02:39:40.661470 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:40.661481 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:40.661541 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:40.661678 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:40.661679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:40.661842 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:40.661892 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:40.661893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:40.662573 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:40.662574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:40.663111 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:40.663113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:40.663518 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:40.663519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:40.663567 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:40.664067 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:40.664071 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:40.665802 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
W20260928 02:39:40.665963 18928 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:40.661429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:55821 (local address 127.9.130.62:34689)
0928 02:39:40.661552 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:40.661554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:40.661562 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:39:40.661585 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:40.661586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:40.661598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:40.661657 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:40.661967 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:40.662451 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:40.663223 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:40.663434 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:40.663498 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:40.665891 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.1:55821: BlockingRecv error: recv got EOF from 127.9.130.1:55821 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
Sep 28 02:39:40 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:39:40 dist-test-slave-kkrh krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.130.2@KRBTEST.COM for oryx/127.9.130.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:39:40.815241 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:40.810727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:34689 (local address 127.9.130.2:36069)
0928 02:39:40.810913 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:40.810923 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:40.811005 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:40.811162 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:40.811163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:40.811309 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:40.811356 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:40.811358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:40.812049 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:40.812050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:40.812600 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:40.812602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:40.812919 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:40.812920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:40.812957 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:40.813443 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:40.813448 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:40.815186 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.130.62:34689: Server oryx/127.9.130.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":121,"threads_started":1}
W20260928 02:39:40.815349 18928 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:40.810832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:36069 (local address 127.9.130.62:34689)
0928 02:39:40.810899 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:40.810902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:40.810992 (+ 90us) server_negotiation.cc:407] Connection header received
0928 02:39:40.811064 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:40.811065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:40.811078 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:40.811142 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:40.811436 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:40.811984 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:40.812674 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:40.812859 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:40.812908 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:40.815278 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.130.2:36069: BlockingRecv error: recv got EOF from 127.9.130.2:36069 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:39:40.872439 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 18451
I20260928 02:39:40.877063 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 18602
I20260928 02:39:40.881130 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 18735
I20260928 02:39:40.885221 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 18380
2026-09-28T02:39:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17535 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T02:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:40Z Disabled control of system clock
I20260928 02:39:40.912657 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:42185
--webserver_interface=127.9.130.62
--webserver_port=0
--builtin_ntp_servers=127.9.130.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:42185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:39:40.991237 18933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:40.991431 18933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:40.991451 18933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:40.992679 18933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:40.992722 18933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:40.992735 18933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:40.992748 18933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:40.992760 18933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:40.994145 18933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:42185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:42185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:40.994432 18933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:40.994738 18933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:41.000134 18941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:41.000098 18939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:41.000125 18933 server_base.cc:1034] running on GCE node
W20260928 02:39:41.000108 18938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:41.000664 18933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:41.001008 18933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:41.002158 18933 hybrid_clock.cc:648] HybridClock initialized: now 1790563181002143 us; error 49 us; skew 500 ppm
I20260928 02:39:41.003473 18933 webserver.cc:460] Webserver started at http://127.9.130.62:36295/ using document root <none> and password file <none>
I20260928 02:39:41.003746 18933 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:41.003816 18933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:41.003985 18933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:41.005029 18933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "adf224096ef7445da9c3d75df72bfa12"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "62981bdd7abe24211a5a0a06819c4892"
server_key_iv: "e9bb338b523a9ec75e9015a780abb2d4"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.005393 18933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "adf224096ef7445da9c3d75df72bfa12"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "62981bdd7abe24211a5a0a06819c4892"
server_key_iv: "e9bb338b523a9ec75e9015a780abb2d4"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.006678 18933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:39:41.007373 18947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.007547 18933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:41.007683 18933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "adf224096ef7445da9c3d75df72bfa12"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "62981bdd7abe24211a5a0a06819c4892"
server_key_iv: "e9bb338b523a9ec75e9015a780abb2d4"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.007759 18933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:41.028643 18933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:41.028965 18933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:41.029120 18933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:41.033195 18933 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:42185
I20260928 02:39:41.033228 18999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:42185 every 8 connection(s)
I20260928 02:39:41.033584 18933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:41.034361 19000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:41.036897 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 18933
I20260928 02:39:41.037009 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:41.037214 9736 external_mini_cluster.cc:1428] Setting key 48b231f750940e0b3070202cabb662b8
I20260928 02:39:41.037609 19000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12: Bootstrap starting.
I20260928 02:39:41.038628 19000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:41.038983 19000 log.cc:824] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:41.039731 19000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12: No bootstrap required, opened a new log
I20260928 02:39:41.041635 19000 raft_consensus.cc:348] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf224096ef7445da9c3d75df72bfa12" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42185 } }
I20260928 02:39:41.041800 19000 raft_consensus.cc:374] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:41.041849 19000 raft_consensus.cc:729] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf224096ef7445da9c3d75df72bfa12, State: Initialized, Role: FOLLOWER
I20260928 02:39:41.042001 19000 consensus_queue.cc:260] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf224096ef7445da9c3d75df72bfa12" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42185 } }
I20260928 02:39:41.042119 19000 raft_consensus.cc:388] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:41.042157 19000 raft_consensus.cc:482] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:41.042212 19000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:41.042795 19000 raft_consensus.cc:504] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf224096ef7445da9c3d75df72bfa12" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42185 } }
I20260928 02:39:41.042927 19000 leader_election.cc:302] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adf224096ef7445da9c3d75df72bfa12; no voters:
I20260928 02:39:41.043169 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:41.037796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58572 (local address 127.9.130.62:42185)
0928 02:39:41.038003 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:41.038014 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:41.038051 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:39:41.038749 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:41.038761 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:41.038861 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:41.039040 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:41.039293 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.040318 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.041111 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.041520 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.041652 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:41.042419 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:41.042422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:41.042423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:41.042638 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:41.042671 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:41.042681 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:41.042756 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:41.042840 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:39:41.042918 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260928 02:39:41.043184 19000 leader_election.cc:288] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:41.043231 19005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:41.043356 19005 raft_consensus.cc:686] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [term 1 LEADER]: Becoming Leader. State: Replica: adf224096ef7445da9c3d75df72bfa12, State: Running, Role: LEADER
I20260928 02:39:41.043462 19005 consensus_queue.cc:237] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf224096ef7445da9c3d75df72bfa12" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42185 } }
I20260928 02:39:41.043730 19000 sys_catalog.cc:564] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:41.043924 19007 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adf224096ef7445da9c3d75df72bfa12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf224096ef7445da9c3d75df72bfa12" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42185 } } }
I20260928 02:39:41.043941 19006 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adf224096ef7445da9c3d75df72bfa12. Latest consensus state: current_term: 1 leader_uuid: "adf224096ef7445da9c3d75df72bfa12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf224096ef7445da9c3d75df72bfa12" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42185 } } }
I20260928 02:39:41.044060 19006 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:41.044284 19009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:41.044287 19007 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:41.044787 19009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:41.046356 19009 catalog_manager.cc:1347] Generated new cluster ID: 36d1d43af8354f5abcd2cb31fb551cc0
I20260928 02:39:41.046404 19009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:39:41.051954 19009 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:41.052416 19009 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:41.064502 19009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adf224096ef7445da9c3d75df72bfa12: Generated new TSK 0
I20260928 02:39:41.064834 19009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:39:41.071877 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42185
--builtin_ntp_servers=127.9.130.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 02:39:41.150658 19024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:41.150847 19024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:41.150867 19024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:41.152024 19024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:41.152063 19024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:41.152088 19024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:41.153364 19024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:41.153594 19024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:41.153892 19024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:41.158869 19030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:41.158883 19029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:41.158970 19032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:41.159227 19024 server_base.cc:1034] running on GCE node
I20260928 02:39:41.159452 19024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:41.159793 19024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:41.161002 19024 hybrid_clock.cc:648] HybridClock initialized: now 1790563181160979 us; error 40 us; skew 500 ppm
I20260928 02:39:41.162308 19024 webserver.cc:460] Webserver started at http://127.9.130.1:38555/ using document root <none> and password file <none>
I20260928 02:39:41.162559 19024 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:41.162658 19024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:41.162813 19024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:41.163939 19024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "51279d50034b4effb594af025f496215"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "97cfbc29936c768e23838678bf4a8c33"
server_key_iv: "31dca4eeffb43155011829f00e57cb4b"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.164292 19024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "51279d50034b4effb594af025f496215"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "97cfbc29936c768e23838678bf4a8c33"
server_key_iv: "31dca4eeffb43155011829f00e57cb4b"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.165706 19024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:41.166461 19038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.166617 19024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:39:41.166714 19024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "51279d50034b4effb594af025f496215"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "97cfbc29936c768e23838678bf4a8c33"
server_key_iv: "31dca4eeffb43155011829f00e57cb4b"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.166782 19024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:41.183378 19024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:41.183696 19024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:41.183915 19024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:41.184146 19024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:41.184473 19024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:41.184526 19024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.184567 19024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:41.184593 19024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.190436 19024 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:38191
I20260928 02:39:41.190483 19151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:38191 every 8 connection(s)
I20260928 02:39:41.190811 19024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:39:41.196132 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:41.191490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:53733 (local address 127.9.130.62:42185)
0928 02:39:41.191535 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:41.191537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:41.191841 (+ 304us) server_negotiation.cc:407] Connection header received
0928 02:39:41.192409 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:41.192410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:41.192427 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:41.192481 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:41.193150 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.193660 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.194653 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.194855 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.194936 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:41.195850 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:41.195852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:41.195852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:41.195858 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:41.195876 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:41.195880 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:41.195933 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:41.196039 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 02:39:41.196069 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:39:41.196365 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:41.191492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:42185 (local address 127.9.130.1:53733)
0928 02:39:41.191696 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:41.191728 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:41.192299 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:41.192551 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:41.192555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:41.192564 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:41.193049 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:41.193054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.193760 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.193761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:41.194555 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:41.194557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.194944 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.194945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:41.195085 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:41.195672 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:41.195672 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:41.195701 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:41.195725 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:41.195940 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:41.195946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:41.195949 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:41.196022 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:39:41.196120 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260928 02:39:41.196587 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 19024
I20260928 02:39:41.196700 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:39:41.196929 9736 external_mini_cluster.cc:1428] Setting key bde59603b9465ca409a9ac529560a619
I20260928 02:39:41.197033 19152 heartbeater.cc:347] Connected to a master server at 127.9.130.62:42185
I20260928 02:39:41.197176 19152 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:41.197376 19152 heartbeater.cc:511] Master 127.9.130.62:42185 requested a full tablet report, sending...
I20260928 02:39:41.197901 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42185
--builtin_ntp_servers=127.9.130.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:39:41.197933 18964 ts_manager.cc:194] Registered new tserver with Master: 51279d50034b4effb594af025f496215 (127.9.130.1:38191)
I20260928 02:39:41.198836 18964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.130.1:53733
W20260928 02:39:41.278451 19155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:41.278652 19155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:41.278676 19155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:41.279793 19155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:41.279856 19155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:41.279891 19155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:41.281193 19155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:41.281368 19155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:41.281603 19155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:41.286463 19161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:41.286504 19163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:41.286743 19160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:41.286605 19155 server_base.cc:1034] running on GCE node
I20260928 02:39:41.286949 19155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:41.287196 19155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:41.288355 19155 hybrid_clock.cc:648] HybridClock initialized: now 1790563181288341 us; error 27 us; skew 500 ppm
I20260928 02:39:41.289526 19155 webserver.cc:460] Webserver started at http://127.9.130.2:38837/ using document root <none> and password file <none>
I20260928 02:39:41.289779 19155 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:41.289849 19155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:41.289999 19155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:41.291023 19155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "4c77bdf4f13d463d9945e86bc77dc34c"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "b38cb787927ae00e96061f9c92c20955"
server_key_iv: "519345b04472a3d2e48b28d141a74e81"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.291393 19155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "4c77bdf4f13d463d9945e86bc77dc34c"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "b38cb787927ae00e96061f9c92c20955"
server_key_iv: "519345b04472a3d2e48b28d141a74e81"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.292797 19155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:39:41.293486 19169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.293648 19155 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:41.293735 19155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "4c77bdf4f13d463d9945e86bc77dc34c"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "b38cb787927ae00e96061f9c92c20955"
server_key_iv: "519345b04472a3d2e48b28d141a74e81"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.293814 19155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:41.313539 19155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:41.313848 19155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:41.314002 19155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:41.314239 19155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:41.314565 19155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:41.314656 19155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.314698 19155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:41.314728 19155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.320565 19155 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:41575
I20260928 02:39:41.320660 19282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:41575 every 8 connection(s)
I20260928 02:39:41.320958 19155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:39:41.322974 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 19155
I20260928 02:39:41.323100 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:39:41.323299 9736 external_mini_cluster.cc:1428] Setting key 99a69dadb850ca24bc2c35b6b8e8237f
I20260928 02:39:41.324321 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42185
--builtin_ntp_servers=127.9.130.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:39:41.326613 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:41.321709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:43083 (local address 127.9.130.62:42185)
0928 02:39:41.321756 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:41.321758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:41.322038 (+ 280us) server_negotiation.cc:407] Connection header received
0928 02:39:41.322564 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:41.322566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:41.322577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:41.322630 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:41.323350 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.324109 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.325149 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.325417 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.325493 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:41.326347 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:41.326349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:41.326349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:41.326354 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:41.326369 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:41.326374 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:41.326433 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:41.326539 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 02:39:41.326558 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:39:41.326889 19285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:41.321701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:42185 (local address 127.9.130.2:43083)
0928 02:39:41.321911 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:41.321943 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:41.322457 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:41.322696 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:41.322702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:41.322715 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:41.323242 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:41.323260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.324198 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.324199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:41.325049 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:41.325051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.325514 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.325515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:41.325645 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:41.326213 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:41.326214 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:41.326241 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:41.326260 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:41.326449 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:41.326454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:41.326456 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:39:41.326532 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:39:41.326639 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260928 02:39:41.327381 19283 heartbeater.cc:347] Connected to a master server at 127.9.130.62:42185
I20260928 02:39:41.327487 19283 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:41.327657 19283 heartbeater.cc:511] Master 127.9.130.62:42185 requested a full tablet report, sending...
I20260928 02:39:41.328022 18964 ts_manager.cc:194] Registered new tserver with Master: 4c77bdf4f13d463d9945e86bc77dc34c (127.9.130.2:41575)
I20260928 02:39:41.328421 18964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.130.2:43083
W20260928 02:39:41.406086 19286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:41.406312 19286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:41.406349 19286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:41.407506 19286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:41.407564 19286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:41.407605 19286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:41.409044 19286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:41.409296 19286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:41.409605 19286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:41.414753 19291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:41.414844 19292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:41.414845 19286 server_base.cc:1034] running on GCE node
W20260928 02:39:41.414754 19294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:41.415257 19286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:41.415524 19286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:41.416703 19286 hybrid_clock.cc:648] HybridClock initialized: now 1790563181416662 us; error 53 us; skew 500 ppm
I20260928 02:39:41.417959 19286 webserver.cc:460] Webserver started at http://127.9.130.3:37353/ using document root <none> and password file <none>
I20260928 02:39:41.418227 19286 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:41.418316 19286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:41.418488 19286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:41.419602 19286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "c82cd0c3a18b426bac6c99f88efb2b1e"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "9a974948949d650758ea568acadd9501"
server_key_iv: "07e159b8f0c728a275e733b8f640e715"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.420010 19286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "c82cd0c3a18b426bac6c99f88efb2b1e"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "9a974948949d650758ea568acadd9501"
server_key_iv: "07e159b8f0c728a275e733b8f640e715"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.421409 19286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:41.422194 19300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.422372 19286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:41.422477 19286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "c82cd0c3a18b426bac6c99f88efb2b1e"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "9a974948949d650758ea568acadd9501"
server_key_iv: "07e159b8f0c728a275e733b8f640e715"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.422565 19286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:41.447994 19286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:41.448357 19286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:41.448530 19286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:41.448805 19286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:41.449184 19286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:41.449246 19286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.449290 19286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:41.449326 19286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.455229 19286 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:45579
I20260928 02:39:41.455322 19413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:45579 every 8 connection(s)
I20260928 02:39:41.455672 19286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:39:41.459580 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 19286
I20260928 02:39:41.459730 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:39:41.460016 9736 external_mini_cluster.cc:1428] Setting key b0bd6362beb74f2d72c07ca0e0f7bf2b
I20260928 02:39:41.461158 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:41.456515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:35437 (local address 127.9.130.62:42185)
0928 02:39:41.456564 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:41.456567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:41.456871 (+ 304us) server_negotiation.cc:407] Connection header received
0928 02:39:41.457312 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:41.457313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:41.457323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:41.457380 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:41.457996 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.458465 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.459433 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.459772 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.459901 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:41.460854 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:41.460856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:41.460856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:41.460861 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:41.460880 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:41.460885 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:41.460940 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:41.461047 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 02:39:41.461073 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:39:41.461367 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:41.456529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:42185 (local address 127.9.130.3:35437)
0928 02:39:41.456723 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:41.456761 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:41.457228 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:41.457431 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:41.457434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:41.457442 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:41.457926 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:41.457930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.458538 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.458539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:41.459312 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:41.459314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:41.459914 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:41.459915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:41.460029 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:41.460645 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:41.460646 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:41.460685 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:41.460707 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:41.460952 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:41.460957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:41.460960 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:41.461032 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:39:41.461121 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260928 02:39:41.461876 19414 heartbeater.cc:347] Connected to a master server at 127.9.130.62:42185
I20260928 02:39:41.461949 19414 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:41.462134 19414 heartbeater.cc:511] Master 127.9.130.62:42185 requested a full tablet report, sending...
I20260928 02:39:41.462508 18964 ts_manager.cc:194] Registered new tserver with Master: c82cd0c3a18b426bac6c99f88efb2b1e (127.9.130.3:45579)
I20260928 02:39:41.462925 18964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.130.3:35437
I20260928 02:39:41.472281 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:39:41.474169 19003 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:41.473625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.9.130.62:42185)
0928 02:39:41.473729 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:41.473731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:41.473812 (+ 81us) server_negotiation.cc:407] Connection header received
0928 02:39:41.473904 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:41.473905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:41.473915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:41.473967 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:41.474099 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58582: BlockingRecv error: recv got EOF from 127.0.0.1:58582 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:39:41.474242 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 19024
I20260928 02:39:41.478699 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 19155
I20260928 02:39:41.482764 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 19286
I20260928 02:39:41.487262 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 18933
2026-09-28T02:39:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (600 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T02:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:41Z Disabled control of system clock
I20260928 02:39:41.512931 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:42455
--webserver_interface=127.9.130.62
--webserver_port=0
--builtin_ntp_servers=127.9.130.20:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:42455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:39:41.590814 19425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:41.591004 19425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:41.591024 19425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:41.592096 19425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:41.592128 19425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:41.592141 19425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:41.592154 19425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:41.592166 19425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:41.593389 19425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:42455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:42455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:41.593567 19425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:41.593793 19425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:41.598750 19433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:41.598825 19430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:41.598876 19425 server_base.cc:1034] running on GCE node
W20260928 02:39:41.598812 19431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:41.599238 19425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:41.599488 19425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:41.600621 19425 hybrid_clock.cc:648] HybridClock initialized: now 1790563181600627 us; error 53 us; skew 500 ppm
I20260928 02:39:41.601858 19425 webserver.cc:460] Webserver started at http://127.9.130.62:39653/ using document root <none> and password file <none>
I20260928 02:39:41.602124 19425 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:41.602170 19425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:41.602334 19425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:41.603396 19425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "09da2e9b678248ad9fd9e6921d69d800"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "f641bef44f763229890ed41e9b8c61f6"
server_key_iv: "fcf1504cf0e5bc679dd3eefa8e32f9f3"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.603752 19425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "09da2e9b678248ad9fd9e6921d69d800"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "f641bef44f763229890ed41e9b8c61f6"
server_key_iv: "fcf1504cf0e5bc679dd3eefa8e32f9f3"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.605008 19425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:41.605762 19439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.605952 19425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:39:41.606020 19425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "09da2e9b678248ad9fd9e6921d69d800"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "f641bef44f763229890ed41e9b8c61f6"
server_key_iv: "fcf1504cf0e5bc679dd3eefa8e32f9f3"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.606109 19425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:41.609321 19425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:41.609568 19425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:41.609696 19425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:41.613476 19425 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:42455
I20260928 02:39:41.613540 19491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:42455 every 8 connection(s)
I20260928 02:39:41.613828 19425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:41.614452 19492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:41.616513 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 19425
I20260928 02:39:41.616624 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:41.616817 9736 external_mini_cluster.cc:1428] Setting key dc6b94de655c1803a324fe34b1a64bdc
I20260928 02:39:41.617857 19492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800: Bootstrap starting.
I20260928 02:39:41.618630 19492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:41.618948 19492 log.cc:824] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:41.619581 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:41.617331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.9.130.62:42455)
0928 02:39:41.617540 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:41.617560 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:41.617593 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:39:41.618147 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:41.618157 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:41.618220 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:41.618405 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:41.619036 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:41.619040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:41.619040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:41.619045 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:41.619092 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:41.619108 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:41.619198 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:41.619272 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:39:41.619350 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260928 02:39:41.619748 19492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800: No bootstrap required, opened a new log
I20260928 02:39:41.621239 19492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09da2e9b678248ad9fd9e6921d69d800" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42455 } }
I20260928 02:39:41.621371 19492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:41.621394 19492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09da2e9b678248ad9fd9e6921d69d800, State: Initialized, Role: FOLLOWER
I20260928 02:39:41.621544 19492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09da2e9b678248ad9fd9e6921d69d800" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42455 } }
I20260928 02:39:41.621634 19492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:41.621658 19492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:41.621732 19492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:41.622522 19492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09da2e9b678248ad9fd9e6921d69d800" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42455 } }
I20260928 02:39:41.622642 19492 leader_election.cc:302] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09da2e9b678248ad9fd9e6921d69d800; no voters:
I20260928 02:39:41.622822 19492 leader_election.cc:288] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:41.622869 19497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:41.623034 19497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [term 1 LEADER]: Becoming Leader. State: Replica: 09da2e9b678248ad9fd9e6921d69d800, State: Running, Role: LEADER
I20260928 02:39:41.623090 19492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:41.623128 19497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09da2e9b678248ad9fd9e6921d69d800" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42455 } }
I20260928 02:39:41.623473 19499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09da2e9b678248ad9fd9e6921d69d800. Latest consensus state: current_term: 1 leader_uuid: "09da2e9b678248ad9fd9e6921d69d800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09da2e9b678248ad9fd9e6921d69d800" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42455 } } }
I20260928 02:39:41.623585 19499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:41.623463 19498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09da2e9b678248ad9fd9e6921d69d800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09da2e9b678248ad9fd9e6921d69d800" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42455 } } }
I20260928 02:39:41.623764 19498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800 [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:41.624008 19504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:39:41.624441 19504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:41.628149 19504 catalog_manager.cc:1347] Generated new cluster ID: 40558f85c8fb4394b5e18dbe87c2897e
I20260928 02:39:41.628201 19504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:39:41.658527 19504 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:41.658733 19504 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:41.671591 19504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09da2e9b678248ad9fd9e6921d69d800: Generated new TSK 0
I20260928 02:39:41.671835 19504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:39:41.681049 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42455
--builtin_ntp_servers=127.9.130.20:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:39:41.760277 19516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:41.760485 19516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:41.760509 19516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:41.761758 19516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:41.761801 19516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:41.761826 19516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:41.763070 19516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:41.763244 19516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:41.763495 19516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:41.768406 19521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:41.768437 19524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:41.768437 19522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:41.768533 19516 server_base.cc:1034] running on GCE node
I20260928 02:39:41.768883 19516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:41.769117 19516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:41.770280 19516 hybrid_clock.cc:648] HybridClock initialized: now 1790563181770254 us; error 55 us; skew 500 ppm
I20260928 02:39:41.771466 19516 webserver.cc:460] Webserver started at http://127.9.130.1:33041/ using document root <none> and password file <none>
I20260928 02:39:41.771696 19516 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:41.771763 19516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:41.771920 19516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:41.772936 19516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "37b35b6d7b374f2997c4b0b3189c5a21"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "3aa890a87e3ce50bc7b513e3afc3fe90"
server_key_iv: "155e31bbedb70b6c919e76ef5e57cde9"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.773299 19516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "37b35b6d7b374f2997c4b0b3189c5a21"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "3aa890a87e3ce50bc7b513e3afc3fe90"
server_key_iv: "155e31bbedb70b6c919e76ef5e57cde9"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.774569 19516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:41.775332 19530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.775489 19516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:41.775573 19516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "37b35b6d7b374f2997c4b0b3189c5a21"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "3aa890a87e3ce50bc7b513e3afc3fe90"
server_key_iv: "155e31bbedb70b6c919e76ef5e57cde9"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.775624 19516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:41.778812 19516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:41.779007 19516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:41.779096 19516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:41.779239 19516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:41.779466 19516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:41.779495 19516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.779515 19516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:41.779526 19516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.784984 19516 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:36449
I20260928 02:39:41.785025 19643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:36449 every 8 connection(s)
I20260928 02:39:41.785321 19516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:39:41.785532 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 19516
I20260928 02:39:41.785625 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:39:41.785831 9736 external_mini_cluster.cc:1428] Setting key 1082ba825416cf21ed9f39c985e9d4ba
I20260928 02:39:41.786794 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42455
--builtin_ntp_servers=127.9.130.20:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:39:41.788863 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:41.786189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:35387 (local address 127.9.130.62:42455)
0928 02:39:41.786239 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:41.786242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:41.786482 (+ 240us) server_negotiation.cc:407] Connection header received
0928 02:39:41.787071 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:41.787072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:41.787078 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:41.787117 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:41.788646 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:41.788647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:41.788648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:41.788651 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:41.788665 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:41.788669 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:41.788716 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:41.788802 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 02:39:41.788814 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:39:41.789127 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:41.786160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:42455 (local address 127.9.130.1:35387)
0928 02:39:41.786343 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:41.786385 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:41.786960 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:41.787180 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:41.787183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:41.787192 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:41.788454 (+ 1262us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:41.788455 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:41.788498 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:41.788537 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:41.788720 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:41.788723 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:41.788726 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:41.788805 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:39:41.788873 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260928 02:39:41.789669 19644 heartbeater.cc:347] Connected to a master server at 127.9.130.62:42455
I20260928 02:39:41.789783 19644 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:41.789938 19644 heartbeater.cc:511] Master 127.9.130.62:42455 requested a full tablet report, sending...
I20260928 02:39:41.790470 19456 ts_manager.cc:194] Registered new tserver with Master: 37b35b6d7b374f2997c4b0b3189c5a21 (127.9.130.1:36449)
W20260928 02:39:41.866693 19647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:41.866911 19647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:41.866948 19647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:41.868136 19647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:41.868189 19647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:41.868232 19647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:41.869539 19647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:41.869788 19647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:41.870064 19647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:41.874963 19655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:41.874946 19652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:41.874977 19653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:41.875182 19647 server_base.cc:1034] running on GCE node
I20260928 02:39:41.875417 19647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:41.875669 19647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:41.876861 19647 hybrid_clock.cc:648] HybridClock initialized: now 1790563181876842 us; error 48 us; skew 500 ppm
I20260928 02:39:41.878328 19647 webserver.cc:460] Webserver started at http://127.9.130.2:42153/ using document root <none> and password file <none>
I20260928 02:39:41.878588 19647 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:41.878656 19647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:41.878856 19647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:41.880088 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "dbdbd6dba54a43c792003f6729808763"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "5847646db9a0b1823fe10712a23e69d4"
server_key_iv: "6790b64c8d455105378c2f91196d0224"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.880446 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "dbdbd6dba54a43c792003f6729808763"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "5847646db9a0b1823fe10712a23e69d4"
server_key_iv: "6790b64c8d455105378c2f91196d0224"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.881821 19647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:39:41.882633 19661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.882827 19647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:41.882930 19647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "dbdbd6dba54a43c792003f6729808763"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "5847646db9a0b1823fe10712a23e69d4"
server_key_iv: "6790b64c8d455105378c2f91196d0224"
server_key_version: "encryptionkey@0"
I20260928 02:39:41.883016 19647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:41.886334 19647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:41.886616 19647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:41.886766 19647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:41.886979 19647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:41.887300 19647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:41.887355 19647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.887394 19647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:41.887420 19647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:41.893064 19647 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:33765
I20260928 02:39:41.893172 19774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:33765 every 8 connection(s)
I20260928 02:39:41.893451 19647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:39:41.896495 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:41.894207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:32937 (local address 127.9.130.62:42455)
0928 02:39:41.894271 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:41.894273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:41.894480 (+ 207us) server_negotiation.cc:407] Connection header received
0928 02:39:41.894992 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:41.894993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:41.894999 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:41.895064 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:41.896192 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:41.896194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:41.896194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:41.896198 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:41.896212 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:41.896217 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:41.896277 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:41.896427 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 02:39:41.896440 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:39:41.896798 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:41.894214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:42455 (local address 127.9.130.2:32937)
0928 02:39:41.894381 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:41.894412 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:41.894884 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:41.895115 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:41.895118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:41.895125 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:41.896040 (+ 915us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:41.896041 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:41.896067 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:41.896092 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:41.896326 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:41.896332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:41.896335 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:41.896430 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:39:41.896482 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260928 02:39:41.897485 19775 heartbeater.cc:347] Connected to a master server at 127.9.130.62:42455
I20260928 02:39:41.897581 19775 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:41.897758 19775 heartbeater.cc:511] Master 127.9.130.62:42455 requested a full tablet report, sending...
I20260928 02:39:41.898242 19456 ts_manager.cc:194] Registered new tserver with Master: dbdbd6dba54a43c792003f6729808763 (127.9.130.2:33765)
I20260928 02:39:41.902137 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 19647
I20260928 02:39:41.902220 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:39:41.902423 9736 external_mini_cluster.cc:1428] Setting key 726d4e47938a9ba815cb2d38881443fe
I20260928 02:39:41.903306 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42455
--builtin_ntp_servers=127.9.130.20:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:39:41.987931 19778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:41.988157 19778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:41.988190 19778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:41.989344 19778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:41.989408 19778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:41.989451 19778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:41.990805 19778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:41.991063 19778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:41.991400 19778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:39:41.996742 19778 server_base.cc:1034] running on GCE node
W20260928 02:39:41.996687 19784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:41.996675 19786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:41.996706 19783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:41.997212 19778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:41.997466 19778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:41.998607 19778 hybrid_clock.cc:648] HybridClock initialized: now 1790563181998594 us; error 32 us; skew 500 ppm
I20260928 02:39:41.999959 19778 webserver.cc:460] Webserver started at http://127.9.130.3:40299/ using document root <none> and password file <none>
I20260928 02:39:42.000211 19778 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:42.000281 19778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:42.000427 19778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:42.001473 19778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "d9e2cbe44e2e4e758aac3f00c5c2d3f1"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "a8e6c1b03577b27b8dc3563f8f1e5ba4"
server_key_iv: "2174e51228d71c33ce4153b3bf04ffc9"
server_key_version: "encryptionkey@0"
I20260928 02:39:42.001839 19778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "d9e2cbe44e2e4e758aac3f00c5c2d3f1"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "a8e6c1b03577b27b8dc3563f8f1e5ba4"
server_key_iv: "2174e51228d71c33ce4153b3bf04ffc9"
server_key_version: "encryptionkey@0"
I20260928 02:39:42.003197 19778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:39:42.004163 19792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:42.004341 19778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:42.004459 19778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "d9e2cbe44e2e4e758aac3f00c5c2d3f1"
format_stamp: "Formatted at 2026-09-28 02:39:41 on dist-test-slave-kkrh"
server_key: "a8e6c1b03577b27b8dc3563f8f1e5ba4"
server_key_iv: "2174e51228d71c33ce4153b3bf04ffc9"
server_key_version: "encryptionkey@0"
I20260928 02:39:42.004556 19778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:42.008415 19778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:42.008705 19778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:42.008870 19778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:42.009106 19778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:42.009478 19778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:42.009548 19778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:42.009593 19778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:42.009630 19778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:42.016179 19778 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:44937
I20260928 02:39:42.016212 19905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:44937 every 8 connection(s)
I20260928 02:39:42.016556 19778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:39:42.017794 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 19778
I20260928 02:39:42.017892 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:39:42.018096 9736 external_mini_cluster.cc:1428] Setting key 82cceb9a1f5d9851a7e97c15a534718e
I20260928 02:39:42.020004 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:42.017335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:45023 (local address 127.9.130.62:42455)
0928 02:39:42.017398 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:42.017400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:42.017661 (+ 261us) server_negotiation.cc:407] Connection header received
0928 02:39:42.018286 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:42.018287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:42.018295 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:42.018367 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:42.019731 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:42.019732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:42.019733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:39:42.019736 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:42.019750 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:39:42.019754 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:42.019809 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:42.019945 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:39:42.019957 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:39:42.020160 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:42.017338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:42455 (local address 127.9.130.3:45023)
0928 02:39:42.017497 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:42.017526 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:42.018164 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:42.018508 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:42.018511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:42.018518 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:42.019584 (+ 1066us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:39:42.019584 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:42.019609 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:39:42.019634 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:42.019849 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:42.019853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:42.019856 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:42.019934 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:39:42.019979 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260928 02:39:42.020591 19906 heartbeater.cc:347] Connected to a master server at 127.9.130.62:42455
I20260928 02:39:42.020684 19906 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:42.020821 19906 heartbeater.cc:511] Master 127.9.130.62:42455 requested a full tablet report, sending...
I20260928 02:39:42.021143 19456 ts_manager.cc:194] Registered new tserver with Master: d9e2cbe44e2e4e758aac3f00c5c2d3f1 (127.9.130.3:44937)
I20260928 02:39:42.029994 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:39:42.031878 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 19516
W20260928 02:39:42.031889 19495 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:39:42.031417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.9.130.62:42455)
0928 02:39:42.031474 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:42.031476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:42.031576 (+ 100us) server_negotiation.cc:407] Connection header received
0928 02:39:42.031638 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:42.031639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:42.031642 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:42.031683 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:42.031790 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33570: BlockingRecv error: recv got EOF from 127.0.0.1:33570 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:39:42.036296 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 19647
I20260928 02:39:42.041217 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 19778
I20260928 02:39:42.045186 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 19425
2026-09-28T02:39:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (557 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:39:42 dist-test-slave-kkrh krb5kdc[19916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:39:42 dist-test-slave-kkrh krb5kdc[19916](info): set up 2 sockets
Sep 28 02:39:42 dist-test-slave-kkrh krb5kdc[19916](info): commencing operation
krb5kdc: starting...
W20260928 02:39:43.072568 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:39:43 dist-test-slave-kkrh krb5kdc[19916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:39:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:39:43.190454 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:33459
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:33459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:43.270259 19929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:43.270449 19929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:43.270470 19929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:43.271601 19929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:39:43.271641 19929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:43.271656 19929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:43.271668 19929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:39:43.271680 19929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:39:43.273445 19929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:33459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:33459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:43.273698 19929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:43.273919 19929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:43.278648 19935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:43.278728 19937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:43.278662 19934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:43.278879 19929 server_base.cc:1034] running on GCE node
I20260928 02:39:43.279210 19929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:43.279516 19929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:43.280689 19929 hybrid_clock.cc:648] HybridClock initialized: now 1790563183280616 us; error 89 us; skew 500 ppm
Sep 28 02:39:43 dist-test-slave-kkrh krb5kdc[19916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563183, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:43.283340 19929 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:39:43.283874 19929 webserver.cc:460] Webserver started at http://127.9.130.62:40055/ using document root <none> and password file <none>
I20260928 02:39:43.284171 19929 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:43.284245 19929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:43.284392 19929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:43.285591 19929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "da80105a2843416bb548ec15852dc7cb"
format_stamp: "Formatted at 2026-09-28 02:39:43 on dist-test-slave-kkrh"
server_key: "6e74a4284c06709dce25d5cf212a4370"
server_key_iv: "809ce1b2bd36eeabec7855a4105a6258"
server_key_version: "encryptionkey@0"
I20260928 02:39:43.285985 19929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "da80105a2843416bb548ec15852dc7cb"
format_stamp: "Formatted at 2026-09-28 02:39:43 on dist-test-slave-kkrh"
server_key: "6e74a4284c06709dce25d5cf212a4370"
server_key_iv: "809ce1b2bd36eeabec7855a4105a6258"
server_key_version: "encryptionkey@0"
I20260928 02:39:43.287400 19929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:39:43.288226 19944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:43.288416 19929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:43.288530 19929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "da80105a2843416bb548ec15852dc7cb"
format_stamp: "Formatted at 2026-09-28 02:39:43 on dist-test-slave-kkrh"
server_key: "6e74a4284c06709dce25d5cf212a4370"
server_key_iv: "809ce1b2bd36eeabec7855a4105a6258"
server_key_version: "encryptionkey@0"
I20260928 02:39:43.288609 19929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:43.306075 19929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:43.308130 19929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:43.308296 19929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:43.312376 19929 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:33459
I20260928 02:39:43.312405 19996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:33459 every 8 connection(s)
I20260928 02:39:43.312767 19929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:39:43.313691 19997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:43.315045 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 19929
I20260928 02:39:43.315177 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:39:43.315382 9736 external_mini_cluster.cc:1428] Setting key 445e8e02662c5ab7e40fffe50b00695a
I20260928 02:39:43.316224 19997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb: Bootstrap starting.
I20260928 02:39:43.316792 19997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:43.317101 19997 log.cc:824] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:43.317795 19997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb: No bootstrap required, opened a new log
I20260928 02:39:43.319517 19997 raft_consensus.cc:348] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da80105a2843416bb548ec15852dc7cb" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33459 } }
I20260928 02:39:43.319674 19997 raft_consensus.cc:374] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:43.319757 19997 raft_consensus.cc:729] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da80105a2843416bb548ec15852dc7cb, State: Initialized, Role: FOLLOWER
I20260928 02:39:43.319977 19997 consensus_queue.cc:260] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da80105a2843416bb548ec15852dc7cb" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33459 } }
I20260928 02:39:43.320091 19997 raft_consensus.cc:388] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:39:43.320135 19997 raft_consensus.cc:482] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:39:43.320183 19997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:43.320717 19997 raft_consensus.cc:504] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da80105a2843416bb548ec15852dc7cb" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33459 } }
I20260928 02:39:43.320834 19997 leader_election.cc:302] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da80105a2843416bb548ec15852dc7cb; no voters:
I20260928 02:39:43.321018 19997 leader_election.cc:288] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:39:43.321091 20002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:43.321295 20002 raft_consensus.cc:686] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [term 1 LEADER]: Becoming Leader. State: Replica: da80105a2843416bb548ec15852dc7cb, State: Running, Role: LEADER
I20260928 02:39:43.321461 20002 consensus_queue.cc:237] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da80105a2843416bb548ec15852dc7cb" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33459 } }
I20260928 02:39:43.321280 19997 sys_catalog.cc:564] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:39:43.321856 20004 sys_catalog.cc:455] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da80105a2843416bb548ec15852dc7cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da80105a2843416bb548ec15852dc7cb" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33459 } } }
I20260928 02:39:43.322122 20004 sys_catalog.cc:458] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:43.322044 20003 sys_catalog.cc:455] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader da80105a2843416bb548ec15852dc7cb. Latest consensus state: current_term: 1 leader_uuid: "da80105a2843416bb548ec15852dc7cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da80105a2843416bb548ec15852dc7cb" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33459 } } }
I20260928 02:39:43.322407 20003 sys_catalog.cc:458] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb [sys.catalog]: This master's current role is: LEADER
I20260928 02:39:43.322593 20007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 02:39:43 dist-test-slave-kkrh krb5kdc[19916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:43.323082 20007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:39:43.325150 20007 catalog_manager.cc:1347] Generated new cluster ID: c774c276edca496d929c898d97d8bcb4
I20260928 02:39:43.325203 20007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:39:43.327232 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.315938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.9.130.62:33459)
0928 02:39:43.316132 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:43.316143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:43.316181 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:39:43.316685 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:43.316693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:43.316743 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:43.316905 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:43.317687 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.318718 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.319564 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.320154 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.320299 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.323304 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:43.323308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:43.323309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:43.323320 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:43.325394 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:43.325743 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:43.325746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:43.325746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:43.325835 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:43.325999 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:43.326000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:43.326000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:43.326303 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:43.326458 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:43.326913 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 02:39:43.327035 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260928 02:39:43.343012 20007 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:39:43.343781 20007 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:39:43.356360 20007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb: Generated new TSK 0
I20260928 02:39:43.356598 20007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:39:43.384554 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:33459
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:43.464274 20023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:43.464493 20023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:43.464530 20023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:43.465731 20023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:43.465795 20023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:43.465837 20023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:39:43.467240 20023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:33459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:43.467502 20023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:43.467725 20023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:43.472551 20029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:43.472533 20031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:43.472558 20028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:43.472972 20023 server_base.cc:1034] running on GCE node
I20260928 02:39:43.473172 20023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:43.473416 20023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:43.474596 20023 hybrid_clock.cc:648] HybridClock initialized: now 1790563183474575 us; error 31 us; skew 500 ppm
Sep 28 02:39:43 dist-test-slave-kkrh krb5kdc[19916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563183, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:43.477088 20023 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:39:43.477543 20023 webserver.cc:460] Webserver started at http://127.9.130.1:40489/ using document root <none> and password file <none>
I20260928 02:39:43.477778 20023 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:43.477823 20023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:43.477957 20023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:43.479048 20023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "40b074e881e24ec5beb6a7812375ddb6"
format_stamp: "Formatted at 2026-09-28 02:39:43 on dist-test-slave-kkrh"
server_key: "ebba38a8f3db10455a48ab9a90eff8ba"
server_key_iv: "77cd784d6a09216d3da268c0071d0a45"
server_key_version: "encryptionkey@0"
I20260928 02:39:43.479411 20023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "40b074e881e24ec5beb6a7812375ddb6"
format_stamp: "Formatted at 2026-09-28 02:39:43 on dist-test-slave-kkrh"
server_key: "ebba38a8f3db10455a48ab9a90eff8ba"
server_key_iv: "77cd784d6a09216d3da268c0071d0a45"
server_key_version: "encryptionkey@0"
I20260928 02:39:43.480791 20023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:39:43.481583 20038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:43.481786 20023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:39:43.481873 20023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "40b074e881e24ec5beb6a7812375ddb6"
format_stamp: "Formatted at 2026-09-28 02:39:43 on dist-test-slave-kkrh"
server_key: "ebba38a8f3db10455a48ab9a90eff8ba"
server_key_iv: "77cd784d6a09216d3da268c0071d0a45"
server_key_version: "encryptionkey@0"
I20260928 02:39:43.481972 20023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:43.500546 20023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:43.502394 20023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:43.502537 20023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:43.502765 20023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:43.503065 20023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:43.503120 20023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:43.503154 20023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:43.503181 20023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:43.509074 20023 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:42085
I20260928 02:39:43.509121 20151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:42085 every 8 connection(s)
I20260928 02:39:43.509465 20023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:39:43 dist-test-slave-kkrh krb5kdc[19916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563183, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:43.518323 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.510156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:49791 (local address 127.9.130.62:33459)
0928 02:39:43.510231 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:43.510233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:43.510398 (+ 165us) server_negotiation.cc:407] Connection header received
0928 02:39:43.510845 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:43.510846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:43.510855 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:43.510935 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:43.511898 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.512400 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.513375 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.513601 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.513672 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.515723 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:43.515724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:43.515724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:43.515731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:43.516996 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:43.517409 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:43.517410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:43.517410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:43.517459 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:43.517721 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:43.517721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:43.517722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:43.517820 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:43.517868 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:43.518188 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 02:39:43.518257 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:39:43.518563 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.510112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:33459 (local address 127.9.130.1:49791)
0928 02:39:43.510275 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:43.510310 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:43.510765 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:43.511010 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:43.511013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:43.511578 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:43.511758 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.511762 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.512488 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.512489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.513272 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.513274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.513698 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.513700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.513805 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.514341 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:43.514349 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:43.515640 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:43.517106 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:43.517111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:43.517113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:43.517341 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:43.517541 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:43.517542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:43.517543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:43.517656 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:43.517904 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:43.517909 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:43.518065 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 02:39:43.518164 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 02:39:43.518288 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260928 02:39:43.519078 20152 heartbeater.cc:347] Connected to a master server at 127.9.130.62:33459
I20260928 02:39:43.519189 20152 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:43.519377 20152 heartbeater.cc:511] Master 127.9.130.62:33459 requested a full tablet report, sending...
I20260928 02:39:43.519662 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 20023
I20260928 02:39:43.519770 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:39:43.519976 19961 ts_manager.cc:194] Registered new tserver with Master: 40b074e881e24ec5beb6a7812375ddb6 (127.9.130.1:42085)
I20260928 02:39:43.520004 9736 external_mini_cluster.cc:1428] Setting key c1901282d9f13a6f706281b0bac5d290
I20260928 02:39:43.520622 19961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:49791
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:39:43.544283 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:33459
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:43.623600 20157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:43.623812 20157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:43.623862 20157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:43.625000 20157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:43.625066 20157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:43.625109 20157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:39:43.626509 20157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:33459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:43.626767 20157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:43.626982 20157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:43.631747 20163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:43.631886 20165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:43.631909 20157 server_base.cc:1034] running on GCE node
W20260928 02:39:43.631984 20162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:43.632279 20157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:43.632519 20157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:43.633683 20157 hybrid_clock.cc:648] HybridClock initialized: now 1790563183633679 us; error 38 us; skew 500 ppm
Sep 28 02:39:43 dist-test-slave-kkrh krb5kdc[19916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563183, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:43.636086 20157 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:39:43.636548 20157 webserver.cc:460] Webserver started at http://127.9.130.2:34991/ using document root <none> and password file <none>
I20260928 02:39:43.636765 20157 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:43.636832 20157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:43.636981 20157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:43.638036 20157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "a9cf077896e94a8891edb0a711d32635"
format_stamp: "Formatted at 2026-09-28 02:39:43 on dist-test-slave-kkrh"
server_key: "ee7297fce322634c5c21ff220104fca3"
server_key_iv: "749ac93096a73079107b00153d931a0b"
server_key_version: "encryptionkey@0"
I20260928 02:39:43.638401 20157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "a9cf077896e94a8891edb0a711d32635"
format_stamp: "Formatted at 2026-09-28 02:39:43 on dist-test-slave-kkrh"
server_key: "ee7297fce322634c5c21ff220104fca3"
server_key_iv: "749ac93096a73079107b00153d931a0b"
server_key_version: "encryptionkey@0"
I20260928 02:39:43.639726 20157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:39:43.640509 20172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:43.640691 20157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:43.640790 20157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "a9cf077896e94a8891edb0a711d32635"
format_stamp: "Formatted at 2026-09-28 02:39:43 on dist-test-slave-kkrh"
server_key: "ee7297fce322634c5c21ff220104fca3"
server_key_iv: "749ac93096a73079107b00153d931a0b"
server_key_version: "encryptionkey@0"
I20260928 02:39:43.640874 20157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:43.659380 20157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:43.661468 20157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:43.661625 20157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:43.661863 20157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:43.662160 20157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:43.662213 20157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:43.662254 20157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:43.662282 20157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:43.668452 20157 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:35877
I20260928 02:39:43.668591 20285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:35877 every 8 connection(s)
I20260928 02:39:43.668849 20157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:39:43 dist-test-slave-kkrh krb5kdc[19916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563183, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:43.677850 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.669590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:42531 (local address 127.9.130.62:33459)
0928 02:39:43.669656 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:43.669658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:43.669854 (+ 196us) server_negotiation.cc:407] Connection header received
0928 02:39:43.670348 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:43.670349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:43.670359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:43.670417 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:43.671267 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.671742 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.672858 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.673036 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.673090 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.675158 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:43.675159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:43.675159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:43.675165 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:43.676483 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:43.676933 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:43.676934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:43.676934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:43.676992 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:43.677244 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:43.677245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:43.677245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:43.677364 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:43.677431 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:43.677718 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 02:39:43.677785 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:39:43.678125 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.669576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:33459 (local address 127.9.130.2:42531)
0928 02:39:43.669732 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:43.669772 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:43.670239 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:43.670469 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:43.670473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:43.671010 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:43.671189 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.671193 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.671935 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.671936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.672770 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.672772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.673115 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.673116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.673213 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.673725 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:43.673734 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:43.675073 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:43.676555 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:43.676561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:43.676563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:43.676791 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:43.677061 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:43.677062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:43.677062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:43.677163 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:43.677444 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:43.677456 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:43.677594 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 02:39:43.677684 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:39:43.677874 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260928 02:39:43.678563 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 20157
I20260928 02:39:43.678689 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:39:43.678790 20286 heartbeater.cc:347] Connected to a master server at 127.9.130.62:33459
I20260928 02:39:43.678901 20286 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:43.678910 9736 external_mini_cluster.cc:1428] Setting key c458bdd6c9084966760bd5082b2ed689
I20260928 02:39:43.679109 20286 heartbeater.cc:511] Master 127.9.130.62:33459 requested a full tablet report, sending...
I20260928 02:39:43.679548 19961 ts_manager.cc:194] Registered new tserver with Master: a9cf077896e94a8891edb0a711d32635 (127.9.130.2:35877)
I20260928 02:39:43.680128 19961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:42531
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:39:43.702834 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:33459
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:39:43.782023 20291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:39:43.782248 20291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:39:43.782284 20291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:39:43.783407 20291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:39:43.783474 20291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:39:43.783525 20291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:39:43.785109 20291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:33459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:39:43.785360 20291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:39:43.785581 20291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:39:43.790617 20297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:43.790723 20291 server_base.cc:1034] running on GCE node
W20260928 02:39:43.790587 20296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:39:43.790587 20299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:39:43.791131 20291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:39:43.791378 20291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:39:43.792538 20291 hybrid_clock.cc:648] HybridClock initialized: now 1790563183792520 us; error 34 us; skew 500 ppm
Sep 28 02:39:43 dist-test-slave-kkrh krb5kdc[19916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563183, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:39:43.794914 20291 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:39:43.795352 20291 webserver.cc:460] Webserver started at http://127.9.130.3:40227/ using document root <none> and password file <none>
I20260928 02:39:43.795578 20291 fs_manager.cc:359] Metadata directory not provided
I20260928 02:39:43.795643 20291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:39:43.795794 20291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:39:43.796811 20291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "7c0353a465004be584b08c3292a6133e"
format_stamp: "Formatted at 2026-09-28 02:39:43 on dist-test-slave-kkrh"
server_key: "9e2e128997f7417c87cb42a72f8bd3e8"
server_key_iv: "4ba00172174502239d3373662cc8283b"
server_key_version: "encryptionkey@0"
I20260928 02:39:43.797168 20291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "7c0353a465004be584b08c3292a6133e"
format_stamp: "Formatted at 2026-09-28 02:39:43 on dist-test-slave-kkrh"
server_key: "9e2e128997f7417c87cb42a72f8bd3e8"
server_key_iv: "4ba00172174502239d3373662cc8283b"
server_key_version: "encryptionkey@0"
I20260928 02:39:43.798465 20291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:39:43.799221 20306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:43.799392 20291 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:39:43.799516 20291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "7c0353a465004be584b08c3292a6133e"
format_stamp: "Formatted at 2026-09-28 02:39:43 on dist-test-slave-kkrh"
server_key: "9e2e128997f7417c87cb42a72f8bd3e8"
server_key_iv: "4ba00172174502239d3373662cc8283b"
server_key_version: "encryptionkey@0"
I20260928 02:39:43.799593 20291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:39:43.808573 20291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:39:43.810503 20291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:39:43.810657 20291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:39:43.810868 20291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:39:43.811174 20291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:39:43.811228 20291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:43.811267 20291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:39:43.811295 20291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:39:43.817019 20291 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:41299
I20260928 02:39:43.817068 20419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:41299 every 8 connection(s)
I20260928 02:39:43.817360 20291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:39:43 dist-test-slave-kkrh krb5kdc[19916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563183, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:39:43.825838 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.818015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:36245 (local address 127.9.130.62:33459)
0928 02:39:43.818102 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:43.818105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:43.818267 (+ 162us) server_negotiation.cc:407] Connection header received
0928 02:39:43.818747 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:43.818748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:43.818759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:43.818813 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:43.819698 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.820301 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.821216 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.821461 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.821510 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.823357 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:43.823359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:43.823359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:43.823365 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:43.824769 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:43.825117 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:43.825118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:43.825118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:43.825166 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:43.825363 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:43.825363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:43.825364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:43.825463 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:43.825511 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:43.825722 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 02:39:43.825778 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:39:43.826089 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.817995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:33459 (local address 127.9.130.3:36245)
0928 02:39:43.818153 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:43.818186 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:43.818639 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:43.818872 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:43.818875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:43.819416 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:39:43.819607 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.819611 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.820377 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.820377 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.821141 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.821143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.821529 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.821530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.821624 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.822014 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:39:43.822021 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:39:43.823281 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:39:43.824838 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:43.824843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:43.824845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:43.825053 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:43.825223 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:39:43.825223 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:39:43.825224 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:39:43.825311 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:39:43.825515 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:39:43.825518 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:39:43.825619 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 02:39:43.825696 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:39:43.825801 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":102,"threads_started":1}
I20260928 02:39:43.826643 20420 heartbeater.cc:347] Connected to a master server at 127.9.130.62:33459
I20260928 02:39:43.826750 20420 heartbeater.cc:464] Registering TS with master...
I20260928 02:39:43.826951 20420 heartbeater.cc:511] Master 127.9.130.62:33459 requested a full tablet report, sending...
I20260928 02:39:43.826987 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 20291
I20260928 02:39:43.827105 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:39:43.827308 9736 external_mini_cluster.cc:1428] Setting key b40438a3bddd6b56ade1688d05a1f9c2
I20260928 02:39:43.827384 19961 ts_manager.cc:194] Registered new tserver with Master: 7c0353a465004be584b08c3292a6133e (127.9.130.3:41299)
I20260928 02:39:43.827872 19961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:36245
I20260928 02:39:43.828176 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:39:43.834498 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.829390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.9.130.62:33459)
0928 02:39:43.829425 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:43.829426 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:43.829519 (+ 93us) server_negotiation.cc:407] Connection header received
0928 02:39:43.829562 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:43.829563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:43.829570 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:43.829621 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:39:43.830079 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.830500 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.831140 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.831303 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.831348 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.832402 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:39:43.832403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:39:43.832403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:39:43.832408 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:39:43.833697 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:43.833971 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:43.833971 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:43.833972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:43.834032 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:39:43.834149 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:39:43.834149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:39:43.834149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:39:43.834254 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:39:43.834325 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:43.834392 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 02:39:43.834441 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:39:43.835867 19960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:39:43.837472 19960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:39:43.843840 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.839650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:42085 (local address 127.0.0.1:39608)
0928 02:39:43.839835 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:43.839875 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:43.839941 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:43.840211 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:43.840213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:43.840215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:43.840299 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.840302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.841485 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.841486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.842733 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.842736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.843467 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.843468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.843554 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.843578 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:39:43.843666 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:39:43.843755 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":57,"threads_started":1}
I20260928 02:39:43.843850 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.839749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39608 (local address 127.9.130.1:42085)
0928 02:39:43.839958 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:43.839962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:43.839973 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:39:43.840078 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:43.840082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:43.840101 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:43.840198 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:43.840384 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.841370 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.842803 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.843384 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.843455 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.843514 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:43.843693 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 02:39:43.843735 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
I20260928 02:39:43.844203 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.839900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.9.130.3:41299)
0928 02:39:43.840381 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:43.840386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:43.840648 (+ 262us) server_negotiation.cc:407] Connection header received
0928 02:39:43.840689 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:43.840691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:43.840705 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:43.840767 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:43.840970 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.841926 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.842979 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.843672 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.843745 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.843796 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:43.844081 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 02:39:43.844128 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":128,"threads_started":1}
I20260928 02:39:43.844046 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.839670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:35877 (local address 127.0.0.1:41780)
0928 02:39:43.840019 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:43.840029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:43.840072 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:43.840478 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:43.840479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:43.840481 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:43.840524 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.840526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.841744 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.841745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.842853 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.842855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.843774 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.843775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.843865 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.843868 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:43.843900 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 02:39:43.843947 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":1,"thread_start_us":46,"threads_started":1}
I20260928 02:39:43.844048 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.839670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:41299 (local address 127.0.0.1:42246)
0928 02:39:43.840429 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:43.840439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:43.840497 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:43.840842 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:43.840843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:43.840845 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:43.840890 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.840891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.842019 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.842020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.842914 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.842916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.843843 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.843844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.843902 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.843905 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:39:43.843949 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:39:43.843978 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":5,"thread_start_us":62,"threads_started":1}
I20260928 02:39:43.844359 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.840150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.9.130.2:35877)
0928 02:39:43.840329 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:43.840334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:43.840344 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:39:43.840382 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:43.840384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:43.840397 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:43.840472 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:43.840592 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.841643 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.842931 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.843673 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.844134 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.844194 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:43.844219 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 02:39:43.844263 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260928 02:39:43.844977 20086 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee2d441a2b642f981b8874027502138 (DEFAULT_TABLE table=test-table [id=d56f206a79e0442f8978f05e2973826a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:43.845083 20220 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee2d441a2b642f981b8874027502138 (DEFAULT_TABLE table=test-table [id=d56f206a79e0442f8978f05e2973826a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:43.845083 20354 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee2d441a2b642f981b8874027502138 (DEFAULT_TABLE table=test-table [id=d56f206a79e0442f8978f05e2973826a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:39:43.845280 20086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee2d441a2b642f981b8874027502138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:43.845376 20354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee2d441a2b642f981b8874027502138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:43.845376 20220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee2d441a2b642f981b8874027502138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:39:43.847767 20434 tablet_bootstrap.cc:492] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6: Bootstrap starting.
I20260928 02:39:43.848393 20434 tablet_bootstrap.cc:654] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:43.848479 20435 tablet_bootstrap.cc:492] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e: Bootstrap starting.
I20260928 02:39:43.848971 20434 log.cc:824] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:43.849167 20435 tablet_bootstrap.cc:654] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:43.849344 20436 tablet_bootstrap.cc:492] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635: Bootstrap starting.
I20260928 02:39:43.849743 20435 log.cc:824] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:43.850006 20436 tablet_bootstrap.cc:654] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635: Neither blocks nor log segments found. Creating new log.
I20260928 02:39:43.850575 20435 tablet_bootstrap.cc:492] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e: No bootstrap required, opened a new log
I20260928 02:39:43.850560 20436 log.cc:824] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635: Log is configured to *not* fsync() on all Append() calls
I20260928 02:39:43.850677 20435 ts_tablet_manager.cc:1397] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:39:43.851058 20434 tablet_bootstrap.cc:492] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6: No bootstrap required, opened a new log
I20260928 02:39:43.851125 20434 ts_tablet_manager.cc:1397] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:39:43.852057 20436 tablet_bootstrap.cc:492] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635: No bootstrap required, opened a new log
I20260928 02:39:43.852139 20436 ts_tablet_manager.cc:1397] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:39:43.852648 20435 raft_consensus.cc:348] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b074e881e24ec5beb6a7812375ddb6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42085 } } peers { permanent_uuid: "7c0353a465004be584b08c3292a6133e" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41299 } } peers { permanent_uuid: "a9cf077896e94a8891edb0a711d32635" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35877 } }
I20260928 02:39:43.852821 20435 raft_consensus.cc:374] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:43.852854 20435 raft_consensus.cc:729] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0353a465004be584b08c3292a6133e, State: Initialized, Role: FOLLOWER
I20260928 02:39:43.853001 20435 consensus_queue.cc:260] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b074e881e24ec5beb6a7812375ddb6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42085 } } peers { permanent_uuid: "7c0353a465004be584b08c3292a6133e" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41299 } } peers { permanent_uuid: "a9cf077896e94a8891edb0a711d32635" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35877 } }
I20260928 02:39:43.853076 20434 raft_consensus.cc:348] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b074e881e24ec5beb6a7812375ddb6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42085 } } peers { permanent_uuid: "7c0353a465004be584b08c3292a6133e" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41299 } } peers { permanent_uuid: "a9cf077896e94a8891edb0a711d32635" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35877 } }
I20260928 02:39:43.853228 20434 raft_consensus.cc:374] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:43.853272 20434 raft_consensus.cc:729] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b074e881e24ec5beb6a7812375ddb6, State: Initialized, Role: FOLLOWER
I20260928 02:39:43.853282 20435 ts_tablet_manager.cc:1428] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:39:43.853387 20420 heartbeater.cc:503] Master 127.9.130.62:33459 was elected leader, sending a full tablet report...
I20260928 02:39:43.853384 20434 consensus_queue.cc:260] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b074e881e24ec5beb6a7812375ddb6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42085 } } peers { permanent_uuid: "7c0353a465004be584b08c3292a6133e" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41299 } } peers { permanent_uuid: "a9cf077896e94a8891edb0a711d32635" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35877 } }
I20260928 02:39:43.853652 20434 ts_tablet_manager.cc:1428] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:39:43.853698 20152 heartbeater.cc:503] Master 127.9.130.62:33459 was elected leader, sending a full tablet report...
I20260928 02:39:43.853951 20436 raft_consensus.cc:348] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b074e881e24ec5beb6a7812375ddb6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42085 } } peers { permanent_uuid: "7c0353a465004be584b08c3292a6133e" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41299 } } peers { permanent_uuid: "a9cf077896e94a8891edb0a711d32635" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35877 } }
I20260928 02:39:43.854161 20436 raft_consensus.cc:374] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:39:43.854276 20436 raft_consensus.cc:729] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9cf077896e94a8891edb0a711d32635, State: Initialized, Role: FOLLOWER
I20260928 02:39:43.854436 20436 consensus_queue.cc:260] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b074e881e24ec5beb6a7812375ddb6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42085 } } peers { permanent_uuid: "7c0353a465004be584b08c3292a6133e" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41299 } } peers { permanent_uuid: "a9cf077896e94a8891edb0a711d32635" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35877 } }
I20260928 02:39:43.854758 20436 ts_tablet_manager.cc:1428] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 02:39:43.854812 20286 heartbeater.cc:503] Master 127.9.130.62:33459 was elected leader, sending a full tablet report...
W20260928 02:39:43.919622 20287 tablet.cc:2369] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:39:43.959535 20440 raft_consensus.cc:482] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:39:43.959707 20440 raft_consensus.cc:504] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b074e881e24ec5beb6a7812375ddb6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42085 } } peers { permanent_uuid: "7c0353a465004be584b08c3292a6133e" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41299 } } peers { permanent_uuid: "a9cf077896e94a8891edb0a711d32635" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35877 } }
I20260928 02:39:43.960191 20440 leader_election.cc:288] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40b074e881e24ec5beb6a7812375ddb6 (127.9.130.1:42085), a9cf077896e94a8891edb0a711d32635 (127.9.130.2:35877)
I20260928 02:39:43.964155 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.960361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:42085 (local address 127.9.130.3:45727)
0928 02:39:43.960414 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:43.960426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:43.960467 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:43.960638 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:43.960639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:43.960641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:43.960714 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.960716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.961674 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.961674 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.963069 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.963072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.963876 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.963877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.963947 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.963953 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:39:43.964015 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:39:43.964060 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":90}
I20260928 02:39:43.964192 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.960469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:60061 (local address 127.9.130.2:35877)
0928 02:39:43.960522 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:43.960523 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:43.960679 (+ 156us) server_negotiation.cc:407] Connection header received
0928 02:39:43.960715 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:43.960716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:43.960726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:43.960789 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:43.960921 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.961599 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.963168 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.963757 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.963861 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.963872 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:43.964077 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:39:43.964122 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:39:43.964192 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.960433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:45727 (local address 127.9.130.1:42085)
0928 02:39:43.960497 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:43.960499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:43.960509 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:39:43.960538 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:43.960540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:43.960550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:43.960617 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:39:43.960796 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.961576 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.963168 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.963757 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.963862 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.963872 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:43.964077 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:39:43.964122 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:39:43.964164 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.960423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:35877 (local address 127.9.130.3:60061)
0928 02:39:43.960582 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:43.960609 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:39:43.960657 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:39:43.960795 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:39:43.960796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:39:43.960797 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:39:43.960857 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.960858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.961673 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.961674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.963084 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:39:43.963086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.963868 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.963869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:39:43.963947 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.963953 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:39:43.964024 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:39:43.964060 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":113,"thread_start_us":112,"threads_started":1}
I20260928 02:39:43.964505 20106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee2d441a2b642f981b8874027502138" candidate_uuid: "7c0353a465004be584b08c3292a6133e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b074e881e24ec5beb6a7812375ddb6" is_pre_election: true
I20260928 02:39:43.964532 20240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee2d441a2b642f981b8874027502138" candidate_uuid: "7c0353a465004be584b08c3292a6133e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9cf077896e94a8891edb0a711d32635" is_pre_election: true
I20260928 02:39:43.964687 20106 raft_consensus.cc:2454] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0353a465004be584b08c3292a6133e in term 0.
I20260928 02:39:43.964687 20240 raft_consensus.cc:2454] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0353a465004be584b08c3292a6133e in term 0.
I20260928 02:39:43.964905 20309 leader_election.cc:302] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40b074e881e24ec5beb6a7812375ddb6, 7c0353a465004be584b08c3292a6133e; no voters:
I20260928 02:39:43.965055 20440 raft_consensus.cc:2781] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:39:43.965149 20440 raft_consensus.cc:482] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:39:43.965183 20440 raft_consensus.cc:3037] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:43.966006 20440 raft_consensus.cc:504] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b074e881e24ec5beb6a7812375ddb6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42085 } } peers { permanent_uuid: "7c0353a465004be584b08c3292a6133e" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41299 } } peers { permanent_uuid: "a9cf077896e94a8891edb0a711d32635" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35877 } }
I20260928 02:39:43.966162 20440 leader_election.cc:288] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [CANDIDATE]: Term 1 election: Requested vote from peers 40b074e881e24ec5beb6a7812375ddb6 (127.9.130.1:42085), a9cf077896e94a8891edb0a711d32635 (127.9.130.2:35877)
I20260928 02:39:43.966342 20240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee2d441a2b642f981b8874027502138" candidate_uuid: "7c0353a465004be584b08c3292a6133e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9cf077896e94a8891edb0a711d32635"
I20260928 02:39:43.966355 20106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee2d441a2b642f981b8874027502138" candidate_uuid: "7c0353a465004be584b08c3292a6133e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b074e881e24ec5beb6a7812375ddb6"
I20260928 02:39:43.966434 20240 raft_consensus.cc:3037] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:43.966426 20106 raft_consensus.cc:3037] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:39:43.967120 20240 raft_consensus.cc:2454] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0353a465004be584b08c3292a6133e in term 1.
I20260928 02:39:43.967121 20106 raft_consensus.cc:2454] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0353a465004be584b08c3292a6133e in term 1.
I20260928 02:39:43.967290 20308 leader_election.cc:302] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0353a465004be584b08c3292a6133e, a9cf077896e94a8891edb0a711d32635; no voters:
I20260928 02:39:43.967391 20440 raft_consensus.cc:2781] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:39:43.967527 20440 raft_consensus.cc:686] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [term 1 LEADER]: Becoming Leader. State: Replica: 7c0353a465004be584b08c3292a6133e, State: Running, Role: LEADER
I20260928 02:39:43.967615 20440 consensus_queue.cc:237] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b074e881e24ec5beb6a7812375ddb6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42085 } } peers { permanent_uuid: "7c0353a465004be584b08c3292a6133e" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41299 } } peers { permanent_uuid: "a9cf077896e94a8891edb0a711d32635" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35877 } }
I20260928 02:39:43.968570 19960 catalog_manager.cc:5495] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c0353a465004be584b08c3292a6133e (127.9.130.3). New cstate: current_term: 1 leader_uuid: "7c0353a465004be584b08c3292a6133e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b074e881e24ec5beb6a7812375ddb6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c0353a465004be584b08c3292a6133e" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9cf077896e94a8891edb0a711d32635" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35877 } health_report { overall_health: UNKNOWN } } }
I20260928 02:39:43.991096 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:43.988670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.9.130.3:41299)
0928 02:39:43.988735 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:39:43.988737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:39:43.988746 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:39:43.988815 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:39:43.988816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:39:43.988837 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:39:43.988929 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:39:43.989082 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.989546 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.990276 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:39:43.990488 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:39:43.990571 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:39:43.990733 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:39:43.990858 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:39:43.990940 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:39:43.990959 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 02:39:43.990987 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:39:43.992831 20240 raft_consensus.cc:1261] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635 [term 1 FOLLOWER]: Refusing update from remote peer 7c0353a465004be584b08c3292a6133e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:43.992843 20106 raft_consensus.cc:1261] T 9ee2d441a2b642f981b8874027502138 P 40b074e881e24ec5beb6a7812375ddb6 [term 1 FOLLOWER]: Refusing update from remote peer 7c0353a465004be584b08c3292a6133e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:39:43.993165 20440 consensus_queue.cc:1035] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [LEADER]: Connected to new peer: Peer: permanent_uuid: "40b074e881e24ec5beb6a7812375ddb6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:43.993319 20444 consensus_queue.cc:1035] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9cf077896e94a8891edb0a711d32635" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:39:43.995602 20447 mvcc.cc:205] Tried to move back new op lower bound from 7334146801632178176 to 7334146801532203008. Current Snapshot: MvccSnapshot[applied={T|T < 7334146801632178176}]
I20260928 02:39:43.996588 20449 mvcc.cc:205] Tried to move back new op lower bound from 7334146801632178176 to 7334146801532203008. Current Snapshot: MvccSnapshot[applied={T|T < 7334146801632178176 or (T in {7334146801632178176})}]
I20260928 02:39:43.996713 20448 mvcc.cc:205] Tried to move back new op lower bound from 7334146801632178176 to 7334146801532203008. Current Snapshot: MvccSnapshot[applied={T|T < 7334146801632178176 or (T in {7334146801632178176})}]
I20260928 02:39:43.998059 19960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:43.998154 19960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:39:43.998842 19960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da80105a2843416bb548ec15852dc7cb: Sending DeleteTablet for 3 replicas of tablet 9ee2d441a2b642f981b8874027502138
I20260928 02:39:43.999187 20354 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee2d441a2b642f981b8874027502138 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:43 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:42246
I20260928 02:39:43.999220 20220 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee2d441a2b642f981b8874027502138 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:43 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:41780
I20260928 02:39:43.999413 20086 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee2d441a2b642f981b8874027502138 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:39:43 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:39608
I20260928 02:39:43.999426 20457 tablet_replica.cc:330] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635: stopping tablet replica
I20260928 02:39:43.999536 20457 raft_consensus.cc:2229] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:39:43.999655 20457 raft_consensus.cc:2258] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:43.999718 20456 tablet_replica.cc:330] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e: stopping tablet replica
I20260928 02:39:43.999886 20456 raft_consensus.cc:2229] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:39:43.999928 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 20023
I20260928 02:39:44.000407 20456 raft_consensus.cc:2258] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:39:44.000633 20457 ts_tablet_manager.cc:1905] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:44.000965 20456 ts_tablet_manager.cc:1905] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:39:44.002281 20457 ts_tablet_manager.cc:1918] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:44.002343 20457 log.cc:1196] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/9ee2d441a2b642f981b8874027502138
I20260928 02:39:44.002627 20457 ts_tablet_manager.cc:1939] T 9ee2d441a2b642f981b8874027502138 P a9cf077896e94a8891edb0a711d32635: Deleting consensus metadata
I20260928 02:39:44.003046 20456 ts_tablet_manager.cc:1918] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:39:44.003105 20456 log.cc:1196] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/9ee2d441a2b642f981b8874027502138
I20260928 02:39:44.003095 19946 catalog_manager.cc:4841] TS a9cf077896e94a8891edb0a711d32635 (127.9.130.2:35877): tablet 9ee2d441a2b642f981b8874027502138 (table test-table [id=d56f206a79e0442f8978f05e2973826a]) successfully deleted
I20260928 02:39:44.003360 20456 ts_tablet_manager.cc:1939] T 9ee2d441a2b642f981b8874027502138 P 7c0353a465004be584b08c3292a6133e: Deleting consensus metadata
I20260928 02:39:44.003923 19948 catalog_manager.cc:4841] TS 7c0353a465004be584b08c3292a6133e (127.9.130.3:41299): tablet 9ee2d441a2b642f981b8874027502138 (table test-table [id=d56f206a79e0442f8978f05e2973826a]) successfully deleted
W20260928 02:39:44.007124 19947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:42085 (error 108)
I20260928 02:39:44.007642 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:39:44.007500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:42085 (local address 127.0.0.1:39614)
0928 02:39:44.007561 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 02:39:44.007605 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 02:39:44.007706 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 20157
W20260928 02:39:44.009298 19947 catalog_manager.cc:4584] TS 40b074e881e24ec5beb6a7812375ddb6 (127.9.130.1:42085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ee2d441a2b642f981b8874027502138: Network error: Client connection negotiation failed: client connection to 127.9.130.1:42085: connect: Connection refused (error 111)
I20260928 02:39:44.012614 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 20291
I20260928 02:39:44.016631 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 19929
2026-09-28T02:39:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1972 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:39:44 dist-test-slave-kkrh krb5kdc[20462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:39:44 dist-test-slave-kkrh krb5kdc[20462](info): set up 2 sockets
Sep 28 02:39:44 dist-test-slave-kkrh krb5kdc[20462](info): commencing operation
krb5kdc: starting...
W20260928 02:39:45.060734 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:39:45 dist-test-slave-kkrh krb5kdc[20462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:39:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/rangeradmin_127.9.130.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/rangeradmin_127.9.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/rangerlookup_127.9.130.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/rangerlookup_127.9.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/HTTP_127.9.130.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/HTTP_127.9.130.20@KRBTEST.COM.keytab.
I20260928 02:39:45.220880 9736 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskL8zBe0/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/postgres -l logfile start
2026-09-28 02:39:47.699 UTC [20492] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:39:47.699 UTC [20492] LOG: listening on IPv4 address "127.9.130.20", port 38309
2026-09-28 02:39:47.703 UTC [20492] LOG: listening on Unix socket "/tmp/.s.PGSQL.38309"
2026-09-28 02:39:47.724 UTC [20495] LOG: database system was shut down at 2026-09-28 02:39:46 UTC
2026-09-28 02:39:47.728 UTC [20492] LOG: database system is ready to accept connections
I20260928 02:39:48.680713 9736 mini_postgres.cc:96] Postgres bound to 38309
2026-09-28 02:39:48.686 UTC [20503] FATAL: database "slave" does not exist
127.9.130.20:38309 - accepting connections
I20260928 02:39:48.687438 9736 mini_ranger.cc:161] Starting Ranger...
I20260928 02:39:48.702008 9736 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:39:49.177173 9736 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:39:49.177256 9736 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-admin
2026-09-28 02:39:49,462 [I] DB FLAVOR :POSTGRES
2026-09-28 02:39:49,463 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:39:49,463 [I] Checking connection..
2026-09-28 02:39:49,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:39:49,700 [I] Checking connection passed.
2026-09-28 02:39:49,700 [I] --------- Verifying version history table ---------
2026-09-28 02:39:49,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:39:49,950 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:39:49,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:39:50,190 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:39:50,190 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:39:50,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:39:50.454 UTC [20567] WARNING: there is no transaction in progress
2026-09-28 02:39:50,460 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:39:50,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:39:50,690 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:39:50,691 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:39:50,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:39:50,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:39:51,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kkrh.c.gcp-upstream.internal','N') ;"
2026-09-28 02:39:51,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:39:51,644 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:39:51,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:39:51,885 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:39:51,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:39:52,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:39:52,368 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:39:52,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:39:53.342 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.358 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.376 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.392 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.411 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.428 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.486 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.493 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.507 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.516 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.527 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.540 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.549 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.554 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.561 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:53.567 UTC [20675] WARNING: there is no transaction in progress
2026-09-28 02:39:54,183 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:39:54,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:39:54,419 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:39:54,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:39:54,680 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:39:54,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:39:54,940 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:39:54,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:39:55,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:39:55,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kkrh.c.gcp-upstream.internal';"
2026-09-28 02:39:55,686 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:39:55,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:39:55,940 [I] DB_PATCHES have already been applied
I20260928 02:39:55.945989 9736 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-admin:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskL8zBe0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:39:55.946048 9736 mini_ranger.cc:193] Using host: 127.9.130.20
I20260928 02:39:55.949018 9736 mini_ranger.cc:227] Ranger admin URL: http://127.9.130.20:36093
Sep 28, 2026 2:39:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:39:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:39:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:39:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:39:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:39:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:39:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:39:57.075043 9736 mini_ranger.cc:160] Time spent starting Ranger: real 8.388s user 0.006s sys 0.003s
Sep 28, 2026 2:40:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:40:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:40:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:40:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:40:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:40:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:40:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:40:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:40:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:40:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:40:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:40:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:40:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:40:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:40:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:40:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:40:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:40:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:40:18.558302 9736 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/rangerkms_127.9.130.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/rangerkms_127.9.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.130.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.130.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.130.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/HTTP_127.9.130.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.130.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/HTTP_127.9.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:40:18 dist-test-slave-kkrh krb5kdc[20462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563218, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:40:18.640142 9736 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:40:18.649197 9736 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:40:19.616418 9736 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:40:19.616503 9736 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-kms
I20260928 02:40:19.616532 9736 mini_ranger_kms.cc:227] Using postgres at 127.9.130.20:38309
2026-09-28 02:40:19,785 [I] DB FLAVOR :POSTGRES
2026-09-28 02:40:19,785 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:40:19,785 [I] Checking connection
2026-09-28 02:40:19,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:40:20,012 [I] connection success
2026-09-28 02:40:20,012 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:40:20,012 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:40:20,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:40:20,238 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:40:20,238 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:40:20,239 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:40:20,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:38309/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:40:20,493 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:40:20.969358 9736 mini_ranger_kms.cc:310] Created kms service
I20260928 02:40:21.196429 9736 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:40:21.243168 9736 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:40:21 dist-test-slave-kkrh krb5kdc[20462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563218, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:40:21.648064 9736 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:40:21.648180 9736 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-kms:/tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:40:21.648202 9736 mini_ranger_kms.cc:240] Using host: 127.9.130.20
I20260928 02:40:21.651706 9736 mini_ranger_kms.cc:276] Ranger KMS PID: 20855
I20260928 02:40:21.651873 9736 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.130.20:46125
Sep 28, 2026 2:40:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:40:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:40:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:40:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:40:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:40:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:40:23.890733 9736 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.251s user 0.010s sys 0.142s
I20260928 02:40:23.890843 9736 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:40:23.890923 9736 mini_ranger_kms.cc:401] 127.9.130.20:46125/kms/v1/keys
Sep 28 02:40:24 dist-test-slave-kkrh krb5kdc[20462](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563224, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.130.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:40:24 dist-test-slave-kkrh krb5kdc[20462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563224, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.130.20@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
Sep 28 02:40:24 dist-test-slave-kkrh krb5kdc[20462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563224, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.130.20@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
Sep 28 02:40:27 dist-test-slave-kkrh krb5kdc[20462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:40:27.099687 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:33853
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:33853
--ranger_config_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.130.20:46125/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:40:27.181063 20906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:40:27.181254 20906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:40:27.181274 20906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:40:27.182392 20906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:40:27.182440 20906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:40:27.182456 20906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:40:27.182467 20906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:40:27.182525 20906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:40:27.184018 20906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.130.20:46125/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:33853
--ranger_config_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:33853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:40:27.184252 20906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:40:27.184486 20906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:40:27.189478 20912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:40:27.189577 20906 server_base.cc:1034] running on GCE node
W20260928 02:40:27.189486 20911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:40:27.189497 20914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:40:27.189924 20906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:40:27.190169 20906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:40:27.191362 20906 hybrid_clock.cc:648] HybridClock initialized: now 1790563227191338 us; error 37 us; skew 500 ppm
Sep 28 02:40:27 dist-test-slave-kkrh krb5kdc[20462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563227, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:40:27.193771 20906 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:40:27.194227 20906 webserver.cc:460] Webserver started at http://127.9.130.62:46855/ using document root <none> and password file <none>
I20260928 02:40:27.194453 20906 fs_manager.cc:359] Metadata directory not provided
I20260928 02:40:27.194514 20906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:40:27.194653 20906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:40:27 dist-test-slave-kkrh krb5kdc[20462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563227, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:40:27.248260 20906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "c3ed4fcf9f6c4787bd19f2842c440b72"
format_stamp: "Formatted at 2026-09-28 02:40:27 on dist-test-slave-kkrh"
server_key: "1e7c1392264976dc069402873a1f0002"
server_key_iv: "e81e49f914323401bc5712329778cc39"
server_key_version: "kuduclusterkey@0"
I20260928 02:40:27.248734 20906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "c3ed4fcf9f6c4787bd19f2842c440b72"
format_stamp: "Formatted at 2026-09-28 02:40:27 on dist-test-slave-kkrh"
server_key: "1e7c1392264976dc069402873a1f0002"
server_key_iv: "e81e49f914323401bc5712329778cc39"
server_key_version: "kuduclusterkey@0"
I20260928 02:40:27.250672 20906 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260928 02:40:27.264958 20921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:40:27.265192 20906 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:40:27.265300 20906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "c3ed4fcf9f6c4787bd19f2842c440b72"
format_stamp: "Formatted at 2026-09-28 02:40:27 on dist-test-slave-kkrh"
server_key: "1e7c1392264976dc069402873a1f0002"
server_key_iv: "e81e49f914323401bc5712329778cc39"
server_key_version: "kuduclusterkey@0"
I20260928 02:40:27.265391 20906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:40:27.289881 20906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:40:27.291925 20906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:40:27.292084 20906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:40:27.297535 20906 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:33853
I20260928 02:40:27.297649 20973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:33853 every 8 connection(s)
I20260928 02:40:27.297952 20906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:40:27.298692 20974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:40:27.300987 20974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72: Bootstrap starting.
I20260928 02:40:27.301558 20974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72: Neither blocks nor log segments found. Creating new log.
I20260928 02:40:27.301846 20974 log.cc:824] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72: Log is configured to *not* fsync() on all Append() calls
I20260928 02:40:27.302579 20974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72: No bootstrap required, opened a new log
I20260928 02:40:27.304617 20974 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ed4fcf9f6c4787bd19f2842c440b72" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33853 } }
I20260928 02:40:27.304781 20974 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:40:27.304831 20974 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ed4fcf9f6c4787bd19f2842c440b72, State: Initialized, Role: FOLLOWER
I20260928 02:40:27.304994 20974 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ed4fcf9f6c4787bd19f2842c440b72" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33853 } }
I20260928 02:40:27.305078 20974 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:40:27.305120 20974 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:40:27.305167 20974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:40:27.305665 20974 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ed4fcf9f6c4787bd19f2842c440b72" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33853 } }
I20260928 02:40:27.305781 20974 leader_election.cc:302] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3ed4fcf9f6c4787bd19f2842c440b72; no voters:
I20260928 02:40:27.305985 20974 leader_election.cc:288] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:40:27.306010 20977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:40:27.306147 20977 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [term 1 LEADER]: Becoming Leader. State: Replica: c3ed4fcf9f6c4787bd19f2842c440b72, State: Running, Role: LEADER
I20260928 02:40:27.306212 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 20906
I20260928 02:40:27.306308 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:40:27.306311 20977 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ed4fcf9f6c4787bd19f2842c440b72" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33853 } }
I20260928 02:40:27.306573 9736 external_mini_cluster.cc:1428] Setting key 345639b80c635cf62cbe28ad10352a28
I20260928 02:40:27.306669 20978 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3ed4fcf9f6c4787bd19f2842c440b72. Latest consensus state: current_term: 1 leader_uuid: "c3ed4fcf9f6c4787bd19f2842c440b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ed4fcf9f6c4787bd19f2842c440b72" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33853 } } }
I20260928 02:40:27.306744 20978 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [sys.catalog]: This master's current role is: LEADER
I20260928 02:40:27.306867 20978 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3ed4fcf9f6c4787bd19f2842c440b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ed4fcf9f6c4787bd19f2842c440b72" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33853 } } }
I20260928 02:40:27.306912 20978 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [sys.catalog]: This master's current role is: LEADER
I20260928 02:40:27.306969 20974 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:40:27.307580 20980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:40:27.308135 20980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:40:27.308457 20974 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 28 02:40:27 dist-test-slave-kkrh krb5kdc[20462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:40:27.320367 20980 catalog_manager.cc:1347] Generated new cluster ID: 69f9d0ef2a274d7ca5952ef7ad688af7
I20260928 02:40:27.320433 20980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:40:27.324160 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:27.307647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.9.130.62:33853)
0928 02:40:27.312720 (+ 5073us) server_negotiation.cc:206] Beginning negotiation
0928 02:40:27.312730 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:40:27.312763 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:40:27.312903 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:40:27.312910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:40:27.312976 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:40:27.314089 (+ 1113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:40:27.314210 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:27.315643 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:27.316760 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:27.317378 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:27.320406 (+ 3028us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:27.320503 (+ 97us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:40:27.320506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:40:27.320506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:40:27.320515 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:40:27.322350 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:40:27.322711 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:40:27.322714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:40:27.322714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:40:27.322798 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:40:27.322960 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:40:27.322960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:40:27.322961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:40:27.323244 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:40:27.323752 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0928 02:40:27.323836 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:40:27.323946 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4994,"thread_start_us":68,"threads_started":1}
I20260928 02:40:27.346562 20980 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:40:27.347390 20980 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:40:27.355597 20980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72: Generated new TSK 0
I20260928 02:40:27.355870 20980 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:40:28.080817 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.130.20:46125/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:33853
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:40:28.188225 21018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:40:28.188546 21018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:40:28.188618 21018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:40:28.190325 21018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:40:28.190377 21018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:40:28.190415 21018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:40:28.192543 21018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.130.20:46125/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:33853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:40:28.192924 21018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:40:28.193169 21018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:40:28.200322 21026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:40:28.200328 21024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:40:28.200366 21023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:40:28.200426 21018 server_base.cc:1034] running on GCE node
I20260928 02:40:28.200843 21018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:40:28.201139 21018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:40:28.202328 21018 hybrid_clock.cc:648] HybridClock initialized: now 1790563228202303 us; error 40 us; skew 500 ppm
Sep 28 02:40:28 dist-test-slave-kkrh krb5kdc[20462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563228, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:40:28.205515 21018 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:40:28.206023 21018 webserver.cc:460] Webserver started at http://127.9.130.1:38643/ using document root <none> and password file <none>
I20260928 02:40:28.206349 21018 fs_manager.cc:359] Metadata directory not provided
I20260928 02:40:28.206426 21018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:40:28.206591 21018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:40:28 dist-test-slave-kkrh krb5kdc[20462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563228, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:40:28.220950 21018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "ab1cc03d6c414c9fb7a990f32f282ed6"
format_stamp: "Formatted at 2026-09-28 02:40:28 on dist-test-slave-kkrh"
server_key: "11b6b43ddb62eece9449c0621a85db30"
server_key_iv: "01c3102fd14e511606d4bff6ca812a6c"
server_key_version: "kuduclusterkey@0"
I20260928 02:40:28.221519 21018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "ab1cc03d6c414c9fb7a990f32f282ed6"
format_stamp: "Formatted at 2026-09-28 02:40:28 on dist-test-slave-kkrh"
server_key: "11b6b43ddb62eece9449c0621a85db30"
server_key_iv: "01c3102fd14e511606d4bff6ca812a6c"
server_key_version: "kuduclusterkey@0"
I20260928 02:40:28.223496 21018 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 02:40:28.240321 21035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:40:28.240607 21018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 02:40:28.240729 21018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "ab1cc03d6c414c9fb7a990f32f282ed6"
format_stamp: "Formatted at 2026-09-28 02:40:28 on dist-test-slave-kkrh"
server_key: "11b6b43ddb62eece9449c0621a85db30"
server_key_iv: "01c3102fd14e511606d4bff6ca812a6c"
server_key_version: "kuduclusterkey@0"
I20260928 02:40:28.240842 21018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:40:28.259443 21018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:40:28.262303 21018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:40:28.262470 21018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:40:28.262748 21018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:40:28.263121 21018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:40:28.263221 21018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:40:28.263325 21018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:40:28.263407 21018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:40:28.270743 21018 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:35145
I20260928 02:40:28.271219 21018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:40:28.273490 21148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:35145 every 8 connection(s)
I20260928 02:40:28.277825 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 21018
I20260928 02:40:28.277922 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:40:28.278126 9736 external_mini_cluster.cc:1428] Setting key 3b9c9e17f148c4e4be63ea4830aff11a
Sep 28 02:40:28 dist-test-slave-kkrh krb5kdc[20462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563228, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
I20260928 02:40:28.291476 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.273073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:33853 (local address 127.9.130.1:33821)
0928 02:40:28.276003 (+ 2930us) negotiation.cc:107] Waiting for socket to connect
0928 02:40:28.276050 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:40:28.276735 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:40:28.280670 (+ 3935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:40:28.280676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:40:28.280959 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:40:28.281222 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.281228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.282217 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.282219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.283476 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.283479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.284084 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.284085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.284227 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.284876 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:40:28.284887 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:40:28.287355 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:40:28.289642 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:40:28.289649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:40:28.289651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:40:28.289957 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:40:28.290290 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:40:28.290291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:40:28.290291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:40:28.290439 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:40:28.290783 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:40:28.290789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:40:28.290961 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 02:40:28.291081 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:40:28.291234 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2886,"thread_start_us":92,"threads_started":1}
I20260928 02:40:28.291915 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.273353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:33821 (local address 127.9.130.62:33853)
0928 02:40:28.280437 (+ 7084us) server_negotiation.cc:206] Beginning negotiation
0928 02:40:28.280439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:40:28.280449 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:40:28.280473 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:40:28.280474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:40:28.280485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:40:28.280565 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:40:28.281339 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.282090 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.283592 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.283920 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.284009 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.287476 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:40:28.287478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:40:28.287478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:40:28.287486 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:40:28.289535 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:40:28.290130 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:40:28.290131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:40:28.290132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:40:28.290197 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:40:28.290519 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:40:28.290520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:40:28.290520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:40:28.290682 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:40:28.290742 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:40:28.291710 (+ 968us) server_negotiation.cc:299] Negotiation successful
0928 02:40:28.291791 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7054,"thread_start_us":73,"threads_started":1}
I20260928 02:40:28.292235 21149 heartbeater.cc:347] Connected to a master server at 127.9.130.62:33853
I20260928 02:40:28.292320 21149 heartbeater.cc:464] Registering TS with master...
I20260928 02:40:28.292492 21149 heartbeater.cc:511] Master 127.9.130.62:33853 requested a full tablet report, sending...
I20260928 02:40:28.292989 20938 ts_manager.cc:194] Registered new tserver with Master: ab1cc03d6c414c9fb7a990f32f282ed6 (127.9.130.1:35145)
I20260928 02:40:28.293875 20938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:33821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:40:28.310741 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.130.20:46125/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:33853
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:40:28 dist-test-slave-kkrh krb5kdc[20462](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563228, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:40:28.428941 21155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:40:28.429217 21155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:40:28.429277 21155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:40:28.430995 21155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:40:28.431093 21155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:40:28.431152 21155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:40:28.433277 21155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.130.20:46125/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:33853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:40:28.433643 21155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:40:28.433911 21155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:40:28.439755 21165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:40:28.439723 21163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:40:28.440024 21162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:40:28.440465 21155 server_base.cc:1034] running on GCE node
I20260928 02:40:28.440690 21155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:40:28.440958 21155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:40:28.442117 21155 hybrid_clock.cc:648] HybridClock initialized: now 1790563228442106 us; error 29 us; skew 500 ppm
Sep 28 02:40:28 dist-test-slave-kkrh krb5kdc[20462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563228, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:40:28.444815 21155 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:40:28.445338 21155 webserver.cc:460] Webserver started at http://127.9.130.2:45977/ using document root <none> and password file <none>
I20260928 02:40:28.445720 21155 fs_manager.cc:359] Metadata directory not provided
I20260928 02:40:28.445825 21155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:40:28.445997 21155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:40:28 dist-test-slave-kkrh krb5kdc[20462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563228, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:40:28.458776 21155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "2a9b3821eddb48c491ad4d5db7eb00d9"
format_stamp: "Formatted at 2026-09-28 02:40:28 on dist-test-slave-kkrh"
server_key: "5cc5b449b054daff46049e7c99a72b83"
server_key_iv: "bb767e1715a4bba2118eaff56d1ca7eb"
server_key_version: "kuduclusterkey@0"
I20260928 02:40:28.459307 21155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "2a9b3821eddb48c491ad4d5db7eb00d9"
format_stamp: "Formatted at 2026-09-28 02:40:28 on dist-test-slave-kkrh"
server_key: "5cc5b449b054daff46049e7c99a72b83"
server_key_iv: "bb767e1715a4bba2118eaff56d1ca7eb"
server_key_version: "kuduclusterkey@0"
I20260928 02:40:28.463745 21155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 02:40:28.476009 21172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:40:28.476286 21155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:40:28.476395 21155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "2a9b3821eddb48c491ad4d5db7eb00d9"
format_stamp: "Formatted at 2026-09-28 02:40:28 on dist-test-slave-kkrh"
server_key: "5cc5b449b054daff46049e7c99a72b83"
server_key_iv: "bb767e1715a4bba2118eaff56d1ca7eb"
server_key_version: "kuduclusterkey@0"
I20260928 02:40:28.476488 21155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:40:28.503290 21155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:40:28.505688 21155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:40:28.505865 21155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:40:28.506109 21155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:40:28.506479 21155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:40:28.506541 21155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:40:28.506577 21155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:40:28.506595 21155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:40:28.514052 21155 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:46441
I20260928 02:40:28.514462 21155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:40:28.521744 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 21155
I20260928 02:40:28.521863 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:40:28.522089 9736 external_mini_cluster.cc:1428] Setting key 76ef9e639a7ef0d56c2eb456b38d01a9
I20260928 02:40:28.523945 21286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:46441 every 8 connection(s)
Sep 28 02:40:28 dist-test-slave-kkrh krb5kdc[20462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563228, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
I20260928 02:40:28.535421 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.524874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:56667 (local address 127.9.130.62:33853)
0928 02:40:28.524917 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:40:28.524919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:40:28.525137 (+ 218us) server_negotiation.cc:407] Connection header received
0928 02:40:28.525779 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:40:28.525780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:40:28.525792 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:40:28.525858 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:40:28.526673 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.527476 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.528994 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.529333 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.529377 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.531995 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:40:28.531997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:40:28.531997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:40:28.532005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:40:28.533732 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:40:28.534268 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:40:28.534269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:40:28.534269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:40:28.534339 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:40:28.534636 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:40:28.534637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:40:28.534638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:40:28.534797 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:40:28.534875 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:40:28.535234 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 02:40:28.535322 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
I20260928 02:40:28.536206 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.524594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:33853 (local address 127.9.130.2:56667)
0928 02:40:28.524986 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 02:40:28.525038 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:40:28.525666 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:40:28.525997 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:40:28.526002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:40:28.526286 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:40:28.526558 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.526564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.527590 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.527590 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.528891 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.528894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.529463 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.529464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.529599 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.530197 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:40:28.530207 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:40:28.531880 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:40:28.533830 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:40:28.533836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:40:28.533838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:40:28.534144 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:40:28.534423 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:40:28.534424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:40:28.534424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:40:28.534559 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:40:28.534878 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:40:28.534884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:40:28.535063 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 02:40:28.535742 (+ 679us) client_negotiation.cc:240] Negotiation successful
0928 02:40:28.535930 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":93,"threads_started":1}
I20260928 02:40:28.536845 21287 heartbeater.cc:347] Connected to a master server at 127.9.130.62:33853
I20260928 02:40:28.536942 21287 heartbeater.cc:464] Registering TS with master...
I20260928 02:40:28.537137 21287 heartbeater.cc:511] Master 127.9.130.62:33853 requested a full tablet report, sending...
I20260928 02:40:28.537518 20938 ts_manager.cc:194] Registered new tserver with Master: 2a9b3821eddb48c491ad4d5db7eb00d9 (127.9.130.2:46441)
I20260928 02:40:28.538007 20938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:56667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:40:28.554597 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.130.20:46125/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:33853
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:40:28.676618 21292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:40:28.676975 21292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:40:28.677068 21292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:40:28.679207 21292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:40:28.679317 21292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:40:28.679402 21292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:40:28.681777 21292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.130.20:46125/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:33853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:40:28.682225 21292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:40:28.682541 21292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:40:28.688695 21298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:40:28.688944 21297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:40:28.689052 21300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:40:28.697242 21292 server_base.cc:1034] running on GCE node
I20260928 02:40:28.697494 21292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:40:28.697768 21292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:40:28.698948 21292 hybrid_clock.cc:648] HybridClock initialized: now 1790563228698928 us; error 35 us; skew 500 ppm
Sep 28 02:40:28 dist-test-slave-kkrh krb5kdc[20462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563228, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:40:28.701637 21292 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:40:28.702071 21292 webserver.cc:460] Webserver started at http://127.9.130.3:43787/ using document root <none> and password file <none>
I20260928 02:40:28.702301 21292 fs_manager.cc:359] Metadata directory not provided
I20260928 02:40:28.702399 21292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:40:28.702553 21292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:40:28 dist-test-slave-kkrh krb5kdc[20462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563228, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:40:28.715917 21292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "dfab73cfdf944c0b9b5f58b76ff91727"
format_stamp: "Formatted at 2026-09-28 02:40:28 on dist-test-slave-kkrh"
server_key: "3554695ad56bec9a29b50cb37b2e3842"
server_key_iv: "701f72e8027f2fc24e8a4d1c5af9e2bc"
server_key_version: "kuduclusterkey@0"
I20260928 02:40:28.716367 21292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "dfab73cfdf944c0b9b5f58b76ff91727"
format_stamp: "Formatted at 2026-09-28 02:40:28 on dist-test-slave-kkrh"
server_key: "3554695ad56bec9a29b50cb37b2e3842"
server_key_iv: "701f72e8027f2fc24e8a4d1c5af9e2bc"
server_key_version: "kuduclusterkey@0"
I20260928 02:40:28.718470 21292 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:40:28.751945 21307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:40:28.752219 21292 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260928 02:40:28.752312 21292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "dfab73cfdf944c0b9b5f58b76ff91727"
format_stamp: "Formatted at 2026-09-28 02:40:28 on dist-test-slave-kkrh"
server_key: "3554695ad56bec9a29b50cb37b2e3842"
server_key_iv: "701f72e8027f2fc24e8a4d1c5af9e2bc"
server_key_version: "kuduclusterkey@0"
I20260928 02:40:28.752409 21292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:40:28.783033 21292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:40:28.789155 21292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:40:28.789395 21292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:40:28.789760 21292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:40:28.790196 21292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:40:28.790288 21292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:40:28.790364 21292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:40:28.790432 21292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:40:28.800603 21292 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:43257
I20260928 02:40:28.801100 21292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:40:28.804453 21420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:43257 every 8 connection(s)
I20260928 02:40:28.807201 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 21292
I20260928 02:40:28.807337 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:40:28.807555 9736 external_mini_cluster.cc:1428] Setting key 1f7e4370ff41c6b0039f269951041268
Sep 28 02:40:28 dist-test-slave-kkrh krb5kdc[20462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563228, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:40:28.816555 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.805015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:41587 (local address 127.9.130.62:33853)
0928 02:40:28.805057 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:40:28.805059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:40:28.805180 (+ 121us) server_negotiation.cc:407] Connection header received
0928 02:40:28.805840 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:40:28.805841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:40:28.805854 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:40:28.805927 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:40:28.806703 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.807553 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.809486 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.809798 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.809869 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.813197 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:40:28.813199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:40:28.813199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:40:28.813206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:40:28.814960 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:40:28.815431 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:40:28.815432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:40:28.815432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:40:28.815497 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:40:28.815799 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:40:28.815800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:40:28.815800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:40:28.815961 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:40:28.816036 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:40:28.816388 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 02:40:28.816466 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:40:28.817045 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.804716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:33853 (local address 127.9.130.3:41587)
0928 02:40:28.805005 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 02:40:28.805049 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:40:28.805707 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:40:28.806000 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:40:28.806004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:40:28.806258 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:40:28.806573 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.806580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.807677 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.807679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.809385 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.809388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.810387 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.810389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.810527 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.811435 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:40:28.811446 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:40:28.813114 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:40:28.815054 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:40:28.815060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:40:28.815062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:40:28.815345 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:40:28.815581 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:40:28.815582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:40:28.815583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:40:28.815717 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:40:28.816038 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:40:28.816044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:40:28.816221 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 02:40:28.816658 (+ 437us) client_negotiation.cc:240] Negotiation successful
0928 02:40:28.816815 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":99,"threads_started":1}
I20260928 02:40:28.817709 21421 heartbeater.cc:347] Connected to a master server at 127.9.130.62:33853
I20260928 02:40:28.817806 21421 heartbeater.cc:464] Registering TS with master...
I20260928 02:40:28.817984 21421 heartbeater.cc:511] Master 127.9.130.62:33853 requested a full tablet report, sending...
I20260928 02:40:28.818360 20938 ts_manager.cc:194] Registered new tserver with Master: dfab73cfdf944c0b9b5f58b76ff91727 (127.9.130.3:43257)
I20260928 02:40:28.819293 20938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:41587
I20260928 02:40:28.819943 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:40:28.829222 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.821459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.9.130.62:33853)
0928 02:40:28.821497 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:40:28.821499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:40:28.821824 (+ 325us) server_negotiation.cc:407] Connection header received
0928 02:40:28.821923 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:40:28.821923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:40:28.821934 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:40:28.821963 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:40:28.822768 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.823484 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.824603 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.824886 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.824930 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.826437 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:40:28.826438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:40:28.826439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:40:28.826446 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:40:28.828045 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:40:28.828441 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:40:28.828442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:40:28.828443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:40:28.828492 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:40:28.828675 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:40:28.828676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:40:28.828676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:40:28.828789 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:40:28.828857 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:40:28.829054 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:40:28.829133 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:40:28.831460 20937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:40:28.833719 20937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:40:28.843552 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.837007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:35145 (local address 127.0.0.1:47320)
0928 02:40:28.837821 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0928 02:40:28.837861 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:40:28.837920 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:40:28.838893 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:40:28.838896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:40:28.838899 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:40:28.838971 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.838973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.840625 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.840626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.842230 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.842232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.843254 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.843255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.843341 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.843354 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:40:28.843421 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:40:28.843471 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":124,"threads_started":1}
I20260928 02:40:28.843889 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.837359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.9.130.1:35145)
0928 02:40:28.838704 (+ 1345us) server_negotiation.cc:206] Beginning negotiation
0928 02:40:28.838708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:40:28.838717 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:40:28.838752 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:40:28.838755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:40:28.838771 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:40:28.838861 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:40:28.839066 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.840525 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.842325 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.843092 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.843164 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.843220 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:40:28.843727 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 02:40:28.843770 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1318,"thread_start_us":56,"threads_started":1}
I20260928 02:40:28.844658 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.837009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:46441 (local address 127.0.0.1:48176)
0928 02:40:28.838950 (+ 1941us) negotiation.cc:107] Waiting for socket to connect
0928 02:40:28.838960 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:40:28.839006 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:40:28.839472 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:40:28.839473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:40:28.839476 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:40:28.839515 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.839517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.840806 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.840807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.842814 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.842816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.844437 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.844438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.844501 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.844504 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:40:28.844550 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:40:28.844585 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1921,"mutex_wait_us":851,"thread_start_us":103,"threads_started":1}
I20260928 02:40:28.844646 21081 tablet_service.cc:1467] Processing CreateTablet for tablet 509ca837dcb14c67a3b6df93b0563f07 (DEFAULT_TABLE table=test-table [id=0ea18c8f334a4059b51eedb23de745c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:40:28.845000 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.837492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.9.130.2:46441)
0928 02:40:28.838636 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0928 02:40:28.838641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:40:28.839062 (+ 421us) server_negotiation.cc:407] Connection header received
0928 02:40:28.839114 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:40:28.839117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:40:28.839131 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:40:28.839198 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:40:28.839601 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.840714 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.842915 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.844273 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.844338 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.844405 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:40:28.844886 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 02:40:28.844925 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":78,"threads_started":1}
I20260928 02:40:28.845003 21081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 509ca837dcb14c67a3b6df93b0563f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:40:28.846340 21221 tablet_service.cc:1467] Processing CreateTablet for tablet 509ca837dcb14c67a3b6df93b0563f07 (DEFAULT_TABLE table=test-table [id=0ea18c8f334a4059b51eedb23de745c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:40:28.846719 21221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 509ca837dcb14c67a3b6df93b0563f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:40:28.848179 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.837244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:43257 (local address 127.0.0.1:44168)
0928 02:40:28.841748 (+ 4504us) negotiation.cc:107] Waiting for socket to connect
0928 02:40:28.841790 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:40:28.841841 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:40:28.842168 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:40:28.842169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:40:28.842171 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:40:28.842206 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.842207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.845503 (+ 3296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.845503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.846639 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.846642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.847666 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.847667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.847723 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.847727 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:40:28.848061 (+ 334us) client_negotiation.cc:240] Negotiation successful
0928 02:40:28.848112 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4489,"thread_start_us":57,"threads_started":1}
I20260928 02:40:28.848444 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.837446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44168 (local address 127.9.130.3:43257)
0928 02:40:28.841954 (+ 4508us) server_negotiation.cc:206] Beginning negotiation
0928 02:40:28.841959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:40:28.841967 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:40:28.842001 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:40:28.842003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:40:28.842017 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:40:28.842085 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:40:28.844067 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.845395 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.846754 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.847489 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.847562 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.847631 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:40:28.848328 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 02:40:28.848368 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4475,"thread_start_us":80,"threads_started":1}
I20260928 02:40:28.849026 21355 tablet_service.cc:1467] Processing CreateTablet for tablet 509ca837dcb14c67a3b6df93b0563f07 (DEFAULT_TABLE table=test-table [id=0ea18c8f334a4059b51eedb23de745c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:40:28.849342 21355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 509ca837dcb14c67a3b6df93b0563f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:40:28.854019 21440 tablet_bootstrap.cc:492] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9: Bootstrap starting.
I20260928 02:40:28.854244 21439 tablet_bootstrap.cc:492] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6: Bootstrap starting.
I20260928 02:40:28.854941 21439 tablet_bootstrap.cc:654] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6: Neither blocks nor log segments found. Creating new log.
I20260928 02:40:28.855093 21440 tablet_bootstrap.cc:654] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9: Neither blocks nor log segments found. Creating new log.
I20260928 02:40:28.855281 21439 log.cc:824] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:40:28.855441 21440 log.cc:824] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:40:28.855666 21441 tablet_bootstrap.cc:492] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: Bootstrap starting.
I20260928 02:40:28.856222 21440 tablet_bootstrap.cc:492] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9: No bootstrap required, opened a new log
I20260928 02:40:28.856281 21440 ts_tablet_manager.cc:1397] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:40:28.856587 21441 tablet_bootstrap.cc:654] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: Neither blocks nor log segments found. Creating new log.
I20260928 02:40:28.856844 21441 log.cc:824] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: Log is configured to *not* fsync() on all Append() calls
I20260928 02:40:28.857539 21439 tablet_bootstrap.cc:492] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6: No bootstrap required, opened a new log
I20260928 02:40:28.857595 21439 ts_tablet_manager.cc:1397] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:40:28.858048 21440 raft_consensus.cc:348] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 35145 } } peers { permanent_uuid: "dfab73cfdf944c0b9b5f58b76ff91727" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43257 } } peers { permanent_uuid: "2a9b3821eddb48c491ad4d5db7eb00d9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46441 } }
I20260928 02:40:28.858204 21440 raft_consensus.cc:374] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:40:28.858232 21440 raft_consensus.cc:729] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9b3821eddb48c491ad4d5db7eb00d9, State: Initialized, Role: FOLLOWER
I20260928 02:40:28.858335 21440 consensus_queue.cc:260] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 35145 } } peers { permanent_uuid: "dfab73cfdf944c0b9b5f58b76ff91727" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43257 } } peers { permanent_uuid: "2a9b3821eddb48c491ad4d5db7eb00d9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46441 } }
I20260928 02:40:28.858541 21440 ts_tablet_manager.cc:1428] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:40:28.858729 21287 heartbeater.cc:503] Master 127.9.130.62:33853 was elected leader, sending a full tablet report...
I20260928 02:40:28.860270 21439 raft_consensus.cc:348] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 35145 } } peers { permanent_uuid: "dfab73cfdf944c0b9b5f58b76ff91727" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43257 } } peers { permanent_uuid: "2a9b3821eddb48c491ad4d5db7eb00d9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46441 } }
I20260928 02:40:28.860431 21439 raft_consensus.cc:374] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:40:28.860469 21439 raft_consensus.cc:729] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1cc03d6c414c9fb7a990f32f282ed6, State: Initialized, Role: FOLLOWER
I20260928 02:40:28.860584 21439 consensus_queue.cc:260] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 35145 } } peers { permanent_uuid: "dfab73cfdf944c0b9b5f58b76ff91727" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43257 } } peers { permanent_uuid: "2a9b3821eddb48c491ad4d5db7eb00d9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46441 } }
I20260928 02:40:28.860834 21439 ts_tablet_manager.cc:1428] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:40:28.860954 21149 heartbeater.cc:503] Master 127.9.130.62:33853 was elected leader, sending a full tablet report...
I20260928 02:40:28.861490 21441 tablet_bootstrap.cc:492] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: No bootstrap required, opened a new log
I20260928 02:40:28.861603 21441 ts_tablet_manager.cc:1397] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 02:40:28.863722 21441 raft_consensus.cc:348] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 35145 } } peers { permanent_uuid: "dfab73cfdf944c0b9b5f58b76ff91727" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43257 } } peers { permanent_uuid: "2a9b3821eddb48c491ad4d5db7eb00d9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46441 } }
I20260928 02:40:28.863938 21441 raft_consensus.cc:374] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:40:28.864007 21441 raft_consensus.cc:729] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfab73cfdf944c0b9b5f58b76ff91727, State: Initialized, Role: FOLLOWER
I20260928 02:40:28.864179 21441 consensus_queue.cc:260] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 35145 } } peers { permanent_uuid: "dfab73cfdf944c0b9b5f58b76ff91727" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43257 } } peers { permanent_uuid: "2a9b3821eddb48c491ad4d5db7eb00d9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46441 } }
I20260928 02:40:28.864455 21441 ts_tablet_manager.cc:1428] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:40:28.864537 21421 heartbeater.cc:503] Master 127.9.130.62:33853 was elected leader, sending a full tablet report...
I20260928 02:40:28.893585 21446 raft_consensus.cc:482] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:40:28.893745 21446 raft_consensus.cc:504] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 35145 } } peers { permanent_uuid: "dfab73cfdf944c0b9b5f58b76ff91727" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43257 } } peers { permanent_uuid: "2a9b3821eddb48c491ad4d5db7eb00d9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46441 } }
I20260928 02:40:28.894169 21446 leader_election.cc:288] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfab73cfdf944c0b9b5f58b76ff91727 (127.9.130.3:43257), 2a9b3821eddb48c491ad4d5db7eb00d9 (127.9.130.2:46441)
Sep 28 02:40:28 dist-test-slave-kkrh krb5kdc[20462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563228, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:40:28.900393 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.894774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:49471 (local address 127.9.130.3:43257)
0928 02:40:28.894822 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:40:28.894824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:40:28.895527 (+ 703us) server_negotiation.cc:407] Connection header received
0928 02:40:28.895661 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:40:28.895662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:40:28.895676 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:40:28.895711 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:40:28.896017 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.896739 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.898948 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.899540 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.900046 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.900061 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 02:40:28.900282 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 02:40:28.900326 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:40:28.900806 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.894445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:43257 (local address 127.9.130.1:49471)
0928 02:40:28.895434 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0928 02:40:28.895447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:40:28.895570 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:40:28.895787 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:40:28.895787 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:40:28.895789 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:40:28.895904 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.895907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.897568 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.897569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.898834 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.898837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.900112 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.900113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.900196 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.900201 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:40:28.900657 (+ 456us) client_negotiation.cc:240] Negotiation successful
0928 02:40:28.900703 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":122,"threads_started":1}
I20260928 02:40:28.901443 21375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "509ca837dcb14c67a3b6df93b0563f07" candidate_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfab73cfdf944c0b9b5f58b76ff91727" is_pre_election: true
I20260928 02:40:28.901638 21375 raft_consensus.cc:2454] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1cc03d6c414c9fb7a990f32f282ed6 in term 0.
I20260928 02:40:28.901883 21038 leader_election.cc:302] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1cc03d6c414c9fb7a990f32f282ed6, dfab73cfdf944c0b9b5f58b76ff91727; no voters:
I20260928 02:40:28.902012 21446 raft_consensus.cc:2781] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:40:28.902053 21446 raft_consensus.cc:482] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:40:28.902078 21446 raft_consensus.cc:3037] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:40:28.903043 21446 raft_consensus.cc:504] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 35145 } } peers { permanent_uuid: "dfab73cfdf944c0b9b5f58b76ff91727" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43257 } } peers { permanent_uuid: "2a9b3821eddb48c491ad4d5db7eb00d9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46441 } }
I20260928 02:40:28.903204 21446 leader_election.cc:288] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [CANDIDATE]: Term 1 election: Requested vote from peers dfab73cfdf944c0b9b5f58b76ff91727 (127.9.130.3:43257), 2a9b3821eddb48c491ad4d5db7eb00d9 (127.9.130.2:46441)
I20260928 02:40:28.903620 21375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "509ca837dcb14c67a3b6df93b0563f07" candidate_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfab73cfdf944c0b9b5f58b76ff91727"
I20260928 02:40:28.903707 21375 raft_consensus.cc:3037] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:40:28.904693 21375 raft_consensus.cc:2454] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1cc03d6c414c9fb7a990f32f282ed6 in term 1.
I20260928 02:40:28.904902 21038 leader_election.cc:302] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1cc03d6c414c9fb7a990f32f282ed6, dfab73cfdf944c0b9b5f58b76ff91727; no voters:
I20260928 02:40:28.905005 21446 raft_consensus.cc:2781] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:40:28.905154 21446 raft_consensus.cc:686] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [term 1 LEADER]: Becoming Leader. State: Replica: ab1cc03d6c414c9fb7a990f32f282ed6, State: Running, Role: LEADER
I20260928 02:40:28.905265 21446 consensus_queue.cc:237] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 35145 } } peers { permanent_uuid: "dfab73cfdf944c0b9b5f58b76ff91727" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43257 } } peers { permanent_uuid: "2a9b3821eddb48c491ad4d5db7eb00d9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46441 } }
I20260928 02:40:28.906370 20937 catalog_manager.cc:5495] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1cc03d6c414c9fb7a990f32f282ed6 (127.9.130.1). New cstate: current_term: 1 leader_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 35145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfab73cfdf944c0b9b5f58b76ff91727" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a9b3821eddb48c491ad4d5db7eb00d9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46441 } health_report { overall_health: UNKNOWN } } }
I20260928 02:40:28.907073 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.895148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:47355 (local address 127.9.130.2:46441)
0928 02:40:28.895192 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:40:28.895194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:40:28.899048 (+ 3854us) server_negotiation.cc:407] Connection header received
0928 02:40:28.899165 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:40:28.899166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:40:28.899177 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:40:28.899212 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:40:28.899935 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.900620 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.906196 (+ 5576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.906691 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.906755 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.906765 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:40:28.906975 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:40:28.907013 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:40:28.907270 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.894956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:46441 (local address 127.9.130.1:47355)
0928 02:40:28.898985 (+ 4029us) negotiation.cc:107] Waiting for socket to connect
0928 02:40:28.898997 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:40:28.899079 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:40:28.899734 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:40:28.899735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:40:28.899737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:40:28.899811 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.899832 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.900815 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.900816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.906084 (+ 5268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:40:28.906087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.906837 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.906837 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:40:28.906901 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.906905 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:40:28.907151 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 02:40:28.907190 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4011,"thread_start_us":82,"threads_started":1}
I20260928 02:40:28.914187 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "509ca837dcb14c67a3b6df93b0563f07" candidate_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a9b3821eddb48c491ad4d5db7eb00d9"
I20260928 02:40:28.914320 21240 raft_consensus.cc:3037] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:40:28.914597 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "509ca837dcb14c67a3b6df93b0563f07" candidate_uuid: "ab1cc03d6c414c9fb7a990f32f282ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a9b3821eddb48c491ad4d5db7eb00d9" is_pre_election: true
I20260928 02:40:28.915333 21240 raft_consensus.cc:2454] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1cc03d6c414c9fb7a990f32f282ed6 in term 1.
I20260928 02:40:28.935232 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.931993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.9.130.1:35145)
0928 02:40:28.932046 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:40:28.932047 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:40:28.932057 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:40:28.932159 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:40:28.932161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:40:28.932186 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:40:28.932269 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:40:28.932421 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.933104 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.934210 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:40:28.934481 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:40:28.934532 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:40:28.934711 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:40:28.934833 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:40:28.934900 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:40:28.935120 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:40:28.935141 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:40:28.937150 21240 raft_consensus.cc:1261] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9 [term 1 FOLLOWER]: Refusing update from remote peer ab1cc03d6c414c9fb7a990f32f282ed6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:40:28.937481 21375 raft_consensus.cc:1261] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727 [term 1 FOLLOWER]: Refusing update from remote peer ab1cc03d6c414c9fb7a990f32f282ed6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:40:28.937853 21446 consensus_queue.cc:1035] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfab73cfdf944c0b9b5f58b76ff91727" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:40:28.938004 21446 consensus_queue.cc:1035] T 509ca837dcb14c67a3b6df93b0563f07 P ab1cc03d6c414c9fb7a990f32f282ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a9b3821eddb48c491ad4d5db7eb00d9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:40:28.941521 21453 mvcc.cc:205] Tried to move back new op lower bound from 7334146985723891712 to 7334146985596665856. Current Snapshot: MvccSnapshot[applied={T|T < 7334146985723891712}]
I20260928 02:40:28.943747 20937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:40:28.943858 20937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:40:28.944593 20937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3ed4fcf9f6c4787bd19f2842c440b72: Sending DeleteTablet for 3 replicas of tablet 509ca837dcb14c67a3b6df93b0563f07
I20260928 02:40:28.944954 21355 tablet_service.cc:1514] Processing DeleteTablet for tablet 509ca837dcb14c67a3b6df93b0563f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:40:28 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:44168
I20260928 02:40:28.944990 21221 tablet_service.cc:1514] Processing DeleteTablet for tablet 509ca837dcb14c67a3b6df93b0563f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:40:28 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:48176
I20260928 02:40:28.945230 21081 tablet_service.cc:1514] Processing DeleteTablet for tablet 509ca837dcb14c67a3b6df93b0563f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:40:28 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:47320
I20260928 02:40:28.945338 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 21018
I20260928 02:40:28.951928 21461 tablet_replica.cc:330] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9: stopping tablet replica
I20260928 02:40:28.951953 21460 tablet_replica.cc:330] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: stopping tablet replica
I20260928 02:40:28.952065 21460 raft_consensus.cc:2229] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:40:28.952173 21461 raft_consensus.cc:2229] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:40:28.952220 21460 pending_rounds.cc:62] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: Trying to abort 2 pending ops.
I20260928 02:40:28.952324 21461 raft_consensus.cc:2258] T 509ca837dcb14c67a3b6df93b0563f07 P 2a9b3821eddb48c491ad4d5db7eb00d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:40:28.952273 21460 pending_rounds.cc:68] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334146985723891712 op_type: WRITE_OP write_request { tablet_id: "509ca837dcb14c67a3b6df93b0563f07" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\310\251\347\325\006\"8\n\ntest-admin\022*\n 0ea18c8f334a4059b51eedb23de745c3\020\001\030\001 \001(\001" signature: "<redacted>""\030\373*\370\261\347\341I\205\027\025\307\316\222Hk\374M&K\"=\263p\340\372_\370%\033n\320\003\347Q\023+\034\227\343\232\213\253H\361R\350\300\365\324\373\360\272 (9\206|\177\244Zf\307\351" signing_key_seq_num: 0 } } request_id { client_id: "4f306f195cff407aa2653a05c0305c7b" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260928 02:40:28.952484 21460 pending_rounds.cc:68] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334146985596665856 op_type: NO_OP noop_request { }
I20260928 02:40:28.952591 21460 raft_consensus.cc:2866] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260928 02:40:28.955864 21460 raft_consensus.cc:2258] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:40:28.956449 21460 ts_tablet_manager.cc:1905] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:40:28.956940 21311 connection.cc:535] server connection from 127.9.130.1:49471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:40:28.957235 20924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:35145 (error 108)
I20260928 02:40:28.958220 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:40:28.958068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:35145 (local address 127.0.0.1:47338)
0928 02:40:28.958124 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 02:40:28.958176 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 02:40:28.958235 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 21155
W20260928 02:40:28.958439 20924 catalog_manager.cc:4584] TS ab1cc03d6c414c9fb7a990f32f282ed6 (127.9.130.1:35145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 509ca837dcb14c67a3b6df93b0563f07: Network error: Client connection negotiation failed: client connection to 127.9.130.1:35145: connect: Connection refused (error 111)
I20260928 02:40:28.959550 21460 ts_tablet_manager.cc:1918] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:40:28.959692 21460 log.cc:1196] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/509ca837dcb14c67a3b6df93b0563f07
I20260928 02:40:28.960397 21460 ts_tablet_manager.cc:1939] T 509ca837dcb14c67a3b6df93b0563f07 P dfab73cfdf944c0b9b5f58b76ff91727: Deleting consensus metadata
I20260928 02:40:28.960933 20924 catalog_manager.cc:4841] TS dfab73cfdf944c0b9b5f58b76ff91727 (127.9.130.3:43257): tablet 509ca837dcb14c67a3b6df93b0563f07 (table test-table [id=0ea18c8f334a4059b51eedb23de745c3]) successfully deleted
W20260928 02:40:28.964715 21434 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:40:28.964465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:46441 (local address 127.0.0.1:48192)
0928 02:40:28.964505 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 02:40:28.964516 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:40:28.964550 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:40:28.964668 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.2:46441: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 02:40:28.965217 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 21292
I20260928 02:40:28.971351 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 20906
I20260928 02:40:28.977644 9736 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:40:29.321754 9736 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:40:29.321861 9736 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:40:29.666328 9736 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:40:29.666 UTC [20492] LOG: received smart shutdown request
2026-09-28 02:40:29.669 UTC [20492] LOG: background worker "logical replication launcher" (PID 20501) exited with exit code 1
2026-09-28 02:40:29.670 UTC [20496] LOG: shutting down
2026-09-28 02:40:29.829 UTC [20492] LOG: database system is shut down
2026-09-28T02:40:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45921 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:40:29 dist-test-slave-kkrh krb5kdc[21473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:40:29 dist-test-slave-kkrh krb5kdc[21473](info): set up 2 sockets
Sep 28 02:40:29 dist-test-slave-kkrh krb5kdc[21473](info): commencing operation
krb5kdc: starting...
W20260928 02:40:31.007768 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:40:31 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:40:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:40:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/rangeradmin_127.9.130.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/rangeradmin_127.9.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/rangerlookup_127.9.130.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/rangerlookup_127.9.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/HTTP_127.9.130.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/HTTP_127.9.130.20@KRBTEST.COM.keytab.
I20260928 02:40:31.164770 9736 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskL8zBe0/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/postgres -l logfile start
2026-09-28 02:40:33.598 UTC [21503] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:40:33.598 UTC [21503] LOG: listening on IPv4 address "127.9.130.20", port 60291
2026-09-28 02:40:33.602 UTC [21503] LOG: listening on Unix socket "/tmp/.s.PGSQL.60291"
2026-09-28 02:40:33.622 UTC [21506] LOG: database system was shut down at 2026-09-28 02:40:32 UTC
2026-09-28 02:40:33.625 UTC [21503] LOG: database system is ready to accept connections
I20260928 02:40:34.575641 9736 mini_postgres.cc:96] Postgres bound to 60291
2026-09-28 02:40:34.581 UTC [21514] FATAL: database "slave" does not exist
127.9.130.20:60291 - accepting connections
I20260928 02:40:34.582211 9736 mini_ranger.cc:161] Starting Ranger...
I20260928 02:40:34.596395 9736 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:40:35.064478 9736 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:40:35.064560 9736 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-admin
2026-09-28 02:40:35,344 [I] DB FLAVOR :POSTGRES
2026-09-28 02:40:35,344 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:40:35,344 [I] Checking connection..
2026-09-28 02:40:35,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:40:35,612 [I] Checking connection passed.
2026-09-28 02:40:35,612 [I] --------- Verifying version history table ---------
2026-09-28 02:40:35,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:40:35,842 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:40:35,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:40:36,066 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:40:36,066 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:40:36,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:40:36.307 UTC [21578] WARNING: there is no transaction in progress
2026-09-28 02:40:36,314 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:40:36,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:40:36,555 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:40:36,555 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:40:36,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:40:36,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:40:37,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kkrh.c.gcp-upstream.internal','N') ;"
2026-09-28 02:40:37,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:40:37,520 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:40:37,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:40:37,745 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:40:37,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:40:37,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:40:38,219 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:40:38,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:40:39.150 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.166 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.181 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.196 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.211 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.228 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.280 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.286 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.299 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.309 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.318 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.332 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.341 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.347 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.353 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39.359 UTC [21686] WARNING: there is no transaction in progress
2026-09-28 02:40:39,962 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:40:39,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:40:40,179 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:40:40,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:40:40,423 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:40:40,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:40:40,667 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:40:40,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:40:40,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:40:41,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kkrh.c.gcp-upstream.internal';"
2026-09-28 02:40:41,418 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:40:41,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:40:41,646 [I] DB_PATCHES have already been applied
I20260928 02:40:41.651511 9736 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-admin:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskL8zBe0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:40:41.651592 9736 mini_ranger.cc:193] Using host: 127.9.130.20
I20260928 02:40:41.654482 9736 mini_ranger.cc:227] Ranger admin URL: http://127.9.130.20:53409
Sep 28, 2026 2:40:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:40:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:40:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:40:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:40:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:40:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:40:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:40:42.741508 9736 mini_ranger.cc:160] Time spent starting Ranger: real 8.159s user 0.001s sys 0.007s
Sep 28, 2026 2:41:03 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:41:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:41:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:41:03 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:41:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:41:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:41:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:41:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:41:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:41:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:41:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:41:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:41:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:41:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:41:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:41:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:41:03 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:41:04 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:41:04.644153 9736 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.130.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.130.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/rangerkms_127.9.130.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.130.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/rangerkms_127.9.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.130.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.130.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.130.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/HTTP_127.9.130.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.130.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/HTTP_127.9.130.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:41:04 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563264, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:41:04.724224 9736 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:41:04.732656 9736 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:41:05.567755 9736 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:41:05.567860 9736 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-kms
I20260928 02:41:05.567893 9736 mini_ranger_kms.cc:227] Using postgres at 127.9.130.20:60291
2026-09-28 02:41:05,724 [I] DB FLAVOR :POSTGRES
2026-09-28 02:41:05,724 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:41:05,724 [I] Checking connection
2026-09-28 02:41:05,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:41:05,950 [I] connection success
2026-09-28 02:41:05,950 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:41:05,950 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:41:05,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:41:06,196 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:41:06,196 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:41:06,197 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:41:06,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.130.20:60291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:41:06,450 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:41:06.911087 9736 mini_ranger_kms.cc:310] Created kms service
I20260928 02:41:07.082580 9736 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:41:07.123939 9736 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:41:07 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563264, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:41:07.497308 9736 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:41:07.497432 9736 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-kms:/tmp/dist-test-taskL8zBe0/build/release/bin/postgresql.jar:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskL8zBe0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:41:07.497452 9736 mini_ranger_kms.cc:240] Using host: 127.9.130.20
I20260928 02:41:07.500758 9736 mini_ranger_kms.cc:276] Ranger KMS PID: 21866
I20260928 02:41:07.500838 9736 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.130.20:41959
Sep 28, 2026 2:41:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:41:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:41:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:41:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:41:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:41:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:41:09.755286 9736 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.031s user 0.005s sys 0.134s
I20260928 02:41:09.755365 9736 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:41:09.755401 9736 mini_ranger_kms.cc:401] 127.9.130.20:41959/kms/v1/keys
Sep 28 02:41:09 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563269, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.130.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:41:10 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563269, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.130.20@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
Sep 28 02:41:10 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563269, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.130.20@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
Sep 28 02:41:12 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:41:12.854930 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:45265
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:45265
--ranger_config_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.130.20:41959/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:12.957365 21917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:12.957563 21917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:12.957584 21917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:12.958776 21917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:12.958817 21917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:12.958832 21917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:12.958843 21917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:12.958855 21917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:12.960523 21917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.130.20:41959/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:45265
--ranger_config_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:45265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:12.960739 21917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:12.960988 21917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:12.967001 21923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:12.967031 21922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:12.967120 21917 server_base.cc:1034] running on GCE node
W20260928 02:41:12.967309 21925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:12.967561 21917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:12.967880 21917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:12.969020 21917 hybrid_clock.cc:648] HybridClock initialized: now 1790563272968995 us; error 37 us; skew 500 ppm
Sep 28 02:41:12 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563272, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:12.977368 21917 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:41:12.977862 21917 webserver.cc:460] Webserver started at http://127.9.130.62:36125/ using document root <none> and password file <none>
I20260928 02:41:12.978099 21917 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:12.978159 21917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:12.978315 21917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:41:12 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563272, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:41:13.023129 21917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "b7200948713d46d78b3958c12c62adae"
format_stamp: "Formatted at 2026-09-28 02:41:13 on dist-test-slave-kkrh"
server_key: "90e76ef18e522000ce38824c0aca0796"
server_key_iv: "08e488352a96face680aa849a3c83376"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:13.023566 21917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "b7200948713d46d78b3958c12c62adae"
format_stamp: "Formatted at 2026-09-28 02:41:13 on dist-test-slave-kkrh"
server_key: "90e76ef18e522000ce38824c0aca0796"
server_key_iv: "08e488352a96face680aa849a3c83376"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:13.025071 21917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:41:13.036273 21932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:13.036485 21917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:41:13.036587 21917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "b7200948713d46d78b3958c12c62adae"
format_stamp: "Formatted at 2026-09-28 02:41:13 on dist-test-slave-kkrh"
server_key: "90e76ef18e522000ce38824c0aca0796"
server_key_iv: "08e488352a96face680aa849a3c83376"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:13.036700 21917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:13.087399 21917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:13.089432 21917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:13.089582 21917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:13.094442 21917 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:45265
I20260928 02:41:13.094467 21984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:45265 every 8 connection(s)
I20260928 02:41:13.094846 21917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:13.095522 21985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:13.098320 21985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae: Bootstrap starting.
I20260928 02:41:13.098944 21985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:13.099239 21985 log.cc:824] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:13.099985 21985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae: No bootstrap required, opened a new log
I20260928 02:41:13.101496 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 21917
I20260928 02:41:13.101599 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:41:13.101821 9736 external_mini_cluster.cc:1428] Setting key bacd44dba4780a2ae412a86620e02dbc
I20260928 02:41:13.102322 21985 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7200948713d46d78b3958c12c62adae" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45265 } }
I20260928 02:41:13.102592 21985 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:13.102674 21985 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7200948713d46d78b3958c12c62adae, State: Initialized, Role: FOLLOWER
I20260928 02:41:13.102825 21985 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7200948713d46d78b3958c12c62adae" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45265 } }
I20260928 02:41:13.102933 21985 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:13.102986 21985 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:13.103055 21985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:13.103588 21985 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7200948713d46d78b3958c12c62adae" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45265 } }
I20260928 02:41:13.103705 21985 leader_election.cc:302] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7200948713d46d78b3958c12c62adae; no voters:
I20260928 02:41:13.103884 21985 leader_election.cc:288] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:13.104127 21985 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:13.105504 21985 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:41:13.105551 21991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:13.105722 21991 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 1 LEADER]: Becoming Leader. State: Replica: b7200948713d46d78b3958c12c62adae, State: Running, Role: LEADER
I20260928 02:41:13.105824 21991 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7200948713d46d78b3958c12c62adae" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45265 } }
I20260928 02:41:13.109325 21991 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7200948713d46d78b3958c12c62adae. Latest consensus state: current_term: 1 leader_uuid: "b7200948713d46d78b3958c12c62adae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7200948713d46d78b3958c12c62adae" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45265 } } }
I20260928 02:41:13.109485 21991 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:13.109723 22006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:13.110225 22006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:41:13 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:13.112011 22002 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7200948713d46d78b3958c12c62adae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7200948713d46d78b3958c12c62adae" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45265 } } }
I20260928 02:41:13.112422 22002 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:13.116108 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:13.102463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.9.130.62:45265)
0928 02:41:13.102643 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:13.102652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:13.102683 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:41:13.102800 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:13.102808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:13.102863 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:13.103012 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:13.103894 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:13.105413 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:13.107219 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:13.108701 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:13.108830 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:13.111733 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:13.111737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:13.111737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:13.111750 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:13.113772 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:13.114206 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:13.114210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:13.114210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:13.114339 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:13.114627 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:13.114628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:13.114628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:13.114991 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:13.115175 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:13.115649 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 02:41:13.115792 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260928 02:41:13.122780 22006 catalog_manager.cc:1347] Generated new cluster ID: 98f31800ca58432c8a26118ea513c944
I20260928 02:41:13.122825 22006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:13.144079 22006 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:13.144937 22006 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:13.151701 22006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae: Generated new TSK 0
I20260928 02:41:13.151996 22006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:41:13.884742 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.130.20:41959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:45265
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:13.998049 22030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:13.998265 22030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:13.998303 22030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:13.999433 22030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:13.999498 22030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:13.999541 22030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:41:14.001127 22030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.130.20:41959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:45265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:14.001372 22030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:14.001593 22030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:14.006927 22036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:14.006879 22038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:14.007299 22030 server_base.cc:1034] running on GCE node
W20260928 02:41:14.007529 22035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:14.007864 22030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:14.008129 22030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:14.009303 22030 hybrid_clock.cc:648] HybridClock initialized: now 1790563274009278 us; error 38 us; skew 500 ppm
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:14.012130 22030 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:41:14.012607 22030 webserver.cc:460] Webserver started at http://127.9.130.1:44341/ using document root <none> and password file <none>
I20260928 02:41:14.012861 22030 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:14.012933 22030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:14.013082 22030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:41:14.027182 22030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "6c19c57215d947f8bd7464eaa2daa084"
format_stamp: "Formatted at 2026-09-28 02:41:14 on dist-test-slave-kkrh"
server_key: "8c64797da1c6cc1603580021285bc5c5"
server_key_iv: "e6c867ec46daae27329af6933f8c004b"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:14.027879 22030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "6c19c57215d947f8bd7464eaa2daa084"
format_stamp: "Formatted at 2026-09-28 02:41:14 on dist-test-slave-kkrh"
server_key: "8c64797da1c6cc1603580021285bc5c5"
server_key_iv: "e6c867ec46daae27329af6933f8c004b"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:14.032223 22030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20260928 02:41:14.048332 22046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:14.048630 22030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 02:41:14.048746 22030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "6c19c57215d947f8bd7464eaa2daa084"
format_stamp: "Formatted at 2026-09-28 02:41:14 on dist-test-slave-kkrh"
server_key: "8c64797da1c6cc1603580021285bc5c5"
server_key_iv: "e6c867ec46daae27329af6933f8c004b"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:14.048834 22030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:14.067206 22030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:14.069769 22030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:14.069970 22030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:14.070256 22030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:14.070654 22030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:14.070714 22030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:14.070784 22030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:14.070822 22030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:14.078609 22030 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:41953
I20260928 02:41:14.078847 22160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:41953 every 8 connection(s)
I20260928 02:41:14.079036 22030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:14.084040 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 22030
I20260928 02:41:14.084156 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:14.084369 9736 external_mini_cluster.cc:1428] Setting key a64e53578bece63c29722a0b0271efef
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:14.090688 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:14.079860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:45265 (local address 127.9.130.1:51393)
0928 02:41:14.080050 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:14.080095 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:14.080624 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:14.081055 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:14.081060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:14.081317 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:14.081612 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:14.081619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:14.082629 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:14.082630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:14.083485 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:14.083487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:14.084053 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:14.084055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:14.084221 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:14.084836 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:14.084846 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:14.086671 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:14.088801 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:14.088808 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:14.088810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:14.089147 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:14.089379 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:14.089390 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:14.089391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:14.089533 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:14.089850 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:14.089855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:14.090056 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 02:41:14.090174 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 02:41:14.090356 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
I20260928 02:41:14.091436 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:14.079899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:51393 (local address 127.9.130.62:45265)
0928 02:41:14.080775 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:14.080777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:14.080784 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:41:14.080807 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:14.080808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:14.080844 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:14.080916 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:14.081735 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:14.082521 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:14.083582 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:14.083920 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:14.084017 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:14.086797 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:14.086798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:14.086798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:14.086804 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:14.088706 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:14.089241 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:14.089243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:14.089243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:14.089308 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:14.089608 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:14.089609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:14.089609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:14.089760 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:14.089823 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:14.091150 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0928 02:41:14.091257 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":90,"threads_started":1}
I20260928 02:41:14.091845 22161 heartbeater.cc:347] Connected to a master server at 127.9.130.62:45265
I20260928 02:41:14.091939 22161 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:14.092129 22161 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
I20260928 02:41:14.092743 21949 ts_manager.cc:194] Registered new tserver with Master: 6c19c57215d947f8bd7464eaa2daa084 (127.9.130.1:41953)
I20260928 02:41:14.093829 21949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:51393
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:41:14.115278 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.130.20:41959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:45265
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:41:14.232905 22167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:14.233237 22167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:14.233322 22167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:14.235153 22167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:14.235260 22167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:14.235343 22167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:41:14.237654 22167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.130.20:41959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:45265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:14.238132 22167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:14.238432 22167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:14.244697 22175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:14.244820 22177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:14.245074 22174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:14.246133 22167 server_base.cc:1034] running on GCE node
I20260928 02:41:14.246326 22167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:14.246588 22167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:14.247768 22167 hybrid_clock.cc:648] HybridClock initialized: now 1790563274247741 us; error 41 us; skew 500 ppm
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:14.250872 22167 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:41:14.251401 22167 webserver.cc:460] Webserver started at http://127.9.130.2:38839/ using document root <none> and password file <none>
I20260928 02:41:14.251647 22167 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:14.251703 22167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:14.251850 22167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:41:14.273633 22167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "5dac3c1d0f66457c9eed52763d160c5e"
format_stamp: "Formatted at 2026-09-28 02:41:14 on dist-test-slave-kkrh"
server_key: "ed8802055140863aec1ec286a6d978e2"
server_key_iv: "4c1b151a7095e491bf9bc529883d43cc"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:14.274204 22167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "5dac3c1d0f66457c9eed52763d160c5e"
format_stamp: "Formatted at 2026-09-28 02:41:14 on dist-test-slave-kkrh"
server_key: "ed8802055140863aec1ec286a6d978e2"
server_key_iv: "4c1b151a7095e491bf9bc529883d43cc"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:14.276729 22167 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:14.289060 22184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:14.290673 22167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:41:14.290793 22167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "5dac3c1d0f66457c9eed52763d160c5e"
format_stamp: "Formatted at 2026-09-28 02:41:14 on dist-test-slave-kkrh"
server_key: "ed8802055140863aec1ec286a6d978e2"
server_key_iv: "4c1b151a7095e491bf9bc529883d43cc"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:14.290886 22167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:14.325253 22167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:14.327880 22167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:14.328028 22167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:14.328301 22167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:14.328706 22167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:14.328765 22167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:14.328804 22167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:14.328876 22167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:14.336138 22167 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:35253
I20260928 02:41:14.336385 22298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:35253 every 8 connection(s)
I20260928 02:41:14.336690 22167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:14.337766 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 22167
I20260928 02:41:14.337888 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:14.338116 9736 external_mini_cluster.cc:1428] Setting key c7a2282f7b6aac10c634e8ac8cf352c8
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
I20260928 02:41:14.350045 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:14.339142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:49133 (local address 127.9.130.62:45265)
0928 02:41:14.339191 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:14.339193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:14.339203 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:41:14.339939 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:14.339940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:14.339953 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:14.340016 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:14.340721 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:14.341517 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:14.343127 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:14.343478 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:14.343546 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:14.346278 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:14.346280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:14.346280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:14.346290 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:14.348187 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:14.348704 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:14.348706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:14.348706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:14.348776 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:14.349103 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:14.349104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:14.349104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:14.349293 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:14.349468 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:14.349794 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 02:41:14.349895 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:41:14.350176 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:14.338845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:45265 (local address 127.9.130.2:49133)
0928 02:41:14.339022 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:14.339076 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:14.339789 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:14.340086 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:14.340090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:14.340341 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:14.340621 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:14.340626 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:14.341633 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:14.341634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:14.343021 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:14.343025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:14.343592 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:14.343594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:14.343759 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:14.344426 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:14.344436 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:14.346152 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:14.348290 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:14.348295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:14.348297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:14.348621 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:14.348866 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:14.348866 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:14.348867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:14.349007 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:14.349408 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:14.349414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:14.349620 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 02:41:14.349740 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:41:14.349948 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
I20260928 02:41:14.351117 22299 heartbeater.cc:347] Connected to a master server at 127.9.130.62:45265
I20260928 02:41:14.351205 22299 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:14.351377 22299 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
I20260928 02:41:14.351769 21949 ts_manager.cc:194] Registered new tserver with Master: 5dac3c1d0f66457c9eed52763d160c5e (127.9.130.2:35253)
I20260928 02:41:14.352283 21949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:49133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:41:14.370651 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.130.20:41959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:45265
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:14.492554 22304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:14.492933 22304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:14.493028 22304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:14.495021 22304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:14.495151 22304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:14.495256 22304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:41:14.497771 22304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.130.20:41959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:45265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:14.498271 22304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:14.498564 22304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:14.504626 22312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:14.504832 22304 server_base.cc:1034] running on GCE node
W20260928 02:41:14.504982 22310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:14.504585 22309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:14.505293 22304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:14.505595 22304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:14.506776 22304 hybrid_clock.cc:648] HybridClock initialized: now 1790563274506758 us; error 37 us; skew 500 ppm
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:14.509837 22304 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:41:14.510257 22304 webserver.cc:460] Webserver started at http://127.9.130.3:40207/ using document root <none> and password file <none>
I20260928 02:41:14.510486 22304 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:14.510553 22304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:14.510706 22304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:41:14.525163 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9"
format_stamp: "Formatted at 2026-09-28 02:41:14 on dist-test-slave-kkrh"
server_key: "5c26d883a4035d27db5433b76378c05f"
server_key_iv: "e0668b056c5250e59af478069d3f99b7"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:14.525743 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9"
format_stamp: "Formatted at 2026-09-28 02:41:14 on dist-test-slave-kkrh"
server_key: "5c26d883a4035d27db5433b76378c05f"
server_key_iv: "e0668b056c5250e59af478069d3f99b7"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:14.530890 22304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260928 02:41:14.553922 22319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:14.554188 22304 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260928 02:41:14.554278 22304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9"
format_stamp: "Formatted at 2026-09-28 02:41:14 on dist-test-slave-kkrh"
server_key: "5c26d883a4035d27db5433b76378c05f"
server_key_iv: "e0668b056c5250e59af478069d3f99b7"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:14.554364 22304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:14.586828 22304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:14.589466 22304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:14.589668 22304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:14.589936 22304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:14.590334 22304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:14.590420 22304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:14.590489 22304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:14.590546 22304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:14.601995 22304 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:43677
I20260928 02:41:14.602571 22304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:14.603150 22432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:43677 every 8 connection(s)
I20260928 02:41:14.609318 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 22304
I20260928 02:41:14.609428 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:14.609620 9736 external_mini_cluster.cc:1428] Setting key 760cf2a98e29770df17e199d4952ea75
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:14.615088 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:14.603852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:45067 (local address 127.9.130.62:45265)
0928 02:41:14.603902 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:14.603904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:14.604191 (+ 287us) server_negotiation.cc:407] Connection header received
0928 02:41:14.604955 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:14.604957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:14.604969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:14.605040 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:14.605828 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:14.606587 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:14.608249 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:14.608557 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:14.608635 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:14.611460 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:14.611462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:14.611462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:14.611470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:14.613369 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:14.613917 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:14.613918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:14.613919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:14.613983 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:14.614311 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:14.614312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:14.614313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:14.614454 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:14.614529 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:14.614909 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 02:41:14.614991 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:41:14.615430 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:14.603843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:45265 (local address 127.9.130.3:45067)
0928 02:41:14.604017 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:14.604070 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:14.604829 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:14.605119 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:14.605124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:14.605393 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:14.605701 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:14.605708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:14.606699 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:14.606701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:14.608150 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:14.608153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:14.608676 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:14.608678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:14.608829 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:14.609503 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:14.609514 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:14.611325 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:14.613469 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:14.613477 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:14.613479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:14.613836 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:14.614068 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:14.614070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:14.614070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:14.614223 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:14.614537 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:14.614544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:14.614738 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 02:41:14.614871 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:41:14.615035 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260928 02:41:14.615957 22433 heartbeater.cc:347] Connected to a master server at 127.9.130.62:45265
I20260928 02:41:14.616065 22433 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:14.616266 22433 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
I20260928 02:41:14.616686 21949 ts_manager.cc:194] Registered new tserver with Master: f8d2ab9f9a4d41d490dfbd9d17168ad9 (127.9.130.3:43677)
I20260928 02:41:14.617220 21949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:45067
I20260928 02:41:14.621130 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:41:14.621884 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 22030
I20260928 02:41:14.628350 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 22167
I20260928 02:41:14.632627 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 22304
I20260928 02:41:14.640357 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 21917
I20260928 02:41:14.647253 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:45265
--webserver_interface=127.9.130.62
--webserver_port=36125
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:45265
--ranger_config_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.130.20:41959/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.130.20:41959/kms with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:14.729040 22436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:14.729229 22436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:14.729250 22436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:14.730298 22436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:14.730330 22436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:14.730345 22436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:14.730356 22436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:14.730368 22436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:14.731802 22436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.130.20:41959/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:45265
--ranger_config_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:45265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=36125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:14.732038 22436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:14.732191 22436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:14.736883 22441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:14.737072 22436 server_base.cc:1034] running on GCE node
W20260928 02:41:14.736881 22442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:14.736897 22444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:14.737452 22436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:14.737721 22436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:14.738886 22436 hybrid_clock.cc:648] HybridClock initialized: now 1790563274738875 us; error 27 us; skew 500 ppm
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:14.741325 22436 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:41:14.741747 22436 webserver.cc:460] Webserver started at http://127.9.130.62:36125/ using document root <none> and password file <none>
I20260928 02:41:14.741986 22436 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:14.742065 22436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:41:14.783385 22436 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:41:14.784188 22452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:14.784407 22436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:14.784510 22436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "b7200948713d46d78b3958c12c62adae"
format_stamp: "Formatted at 2026-09-28 02:41:13 on dist-test-slave-kkrh"
server_key: "90e76ef18e522000ce38824c0aca0796"
server_key_iv: "08e488352a96face680aa849a3c83376"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:14.784907 22436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:14.814931 22436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:14.816854 22436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:14.816998 22436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:14.821072 22436 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:45265
I20260928 02:41:14.821120 22504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:45265 every 8 connection(s)
I20260928 02:41:14.821468 22436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:14.822640 22505 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:41:14.823006 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 22436
I20260928 02:41:14.823294 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.130.20:41959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:41953
--local_ip_for_outbound_sockets=127.9.130.1
--tserver_master_addrs=127.9.130.62:45265
--webserver_port=44341
--webserver_interface=127.9.130.1
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.130.20:41959/kms with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:41:14.823351 22505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae: Bootstrap starting.
I20260928 02:41:14.831166 22505 log.cc:824] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:14.832907 22505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:41:14.833199 22505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae: Bootstrap complete.
I20260928 02:41:14.835474 22505 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7200948713d46d78b3958c12c62adae" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45265 } }
I20260928 02:41:14.835729 22505 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7200948713d46d78b3958c12c62adae, State: Initialized, Role: FOLLOWER
I20260928 02:41:14.835965 22505 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7200948713d46d78b3958c12c62adae" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45265 } }
I20260928 02:41:14.836064 22505 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:14.836118 22505 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:14.836165 22505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:41:14.837147 22505 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7200948713d46d78b3958c12c62adae" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45265 } }
I20260928 02:41:14.837266 22505 leader_election.cc:302] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7200948713d46d78b3958c12c62adae; no voters:
I20260928 02:41:14.837467 22505 leader_election.cc:288] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:41:14.837502 22509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:41:14.837604 22509 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [term 2 LEADER]: Becoming Leader. State: Replica: b7200948713d46d78b3958c12c62adae, State: Running, Role: LEADER
I20260928 02:41:14.837742 22505 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:14.837764 22509 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7200948713d46d78b3958c12c62adae" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45265 } }
I20260928 02:41:14.838163 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7200948713d46d78b3958c12c62adae. Latest consensus state: current_term: 2 leader_uuid: "b7200948713d46d78b3958c12c62adae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7200948713d46d78b3958c12c62adae" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45265 } } }
I20260928 02:41:14.838249 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:14.838467 22510 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b7200948713d46d78b3958c12c62adae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7200948713d46d78b3958c12c62adae" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45265 } } }
I20260928 02:41:14.838546 22510 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:14.839262 22505 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:41:14.841396 22515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:14.842502 22515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:14.842760 22515 catalog_manager.cc:1259] Loaded cluster ID: 98f31800ca58432c8a26118ea513c944
I20260928 02:41:14.842809 22515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:14.844532 22515 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:14.844753 22515 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae: Loaded TSK: 0
I20260928 02:41:14.845012 22515 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:41:14.933311 22507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:14.933574 22507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:14.933609 22507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:14.935312 22507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:14.935362 22507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:14.935402 22507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:41:14.937577 22507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.130.20:41959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:41953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=44341
--tserver_master_addrs=127.9.130.62:45265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:14.937943 22507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:14.938201 22507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:14.944446 22543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:14.944730 22545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:14.945055 22542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:14.945155 22507 server_base.cc:1034] running on GCE node
I20260928 02:41:14.945367 22507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:14.945633 22507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:14.946798 22507 hybrid_clock.cc:648] HybridClock initialized: now 1790563274946785 us; error 34 us; skew 500 ppm
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:14.950049 22507 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:41:14.950604 22507 webserver.cc:460] Webserver started at http://127.9.130.1:44341/ using document root <none> and password file <none>
I20260928 02:41:14.950872 22507 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:14.950951 22507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:41:14 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:41:14.984277 22507 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260928 02:41:14.986146 22553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:14.986374 22507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:14.986466 22507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "6c19c57215d947f8bd7464eaa2daa084"
format_stamp: "Formatted at 2026-09-28 02:41:14 on dist-test-slave-kkrh"
server_key: "8c64797da1c6cc1603580021285bc5c5"
server_key_iv: "e6c867ec46daae27329af6933f8c004b"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:14.986871 22507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:14.998965 22507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:15.001192 22507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:15.001344 22507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:15.001612 22507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:15.001981 22507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:15.002038 22507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:15.002079 22507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:15.002105 22507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:15.008858 22507 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:41953
I20260928 02:41:15.008929 22666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:41953 every 8 connection(s)
I20260928 02:41:15.009289 22507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:15.015250 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 22507
I20260928 02:41:15.015506 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.130.20:41959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:35253
--local_ip_for_outbound_sockets=127.9.130.2
--tserver_master_addrs=127.9.130.62:45265
--webserver_port=38839
--webserver_interface=127.9.130.2
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.130.20:41959/kms with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:41:15 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:15.020560 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:15.009925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:45265 (local address 127.9.130.1:49899)
0928 02:41:15.010097 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:15.010142 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:15.010748 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:15.011209 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:15.011213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:15.011424 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:15.011712 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:15.011719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.012841 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.012842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:15.014107 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:15.014111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.014697 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.014698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:15.014840 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:15.015430 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:15.015440 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:15.017070 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:15.018933 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:15.018939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:15.018940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:15.019156 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:15.019417 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:15.019418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:15.019418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:15.019530 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:15.019942 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:15.019947 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:15.020080 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 02:41:15.020186 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:41:15.020336 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260928 02:41:15.020974 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:15.010086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:49899 (local address 127.9.130.62:45265)
0928 02:41:15.010257 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:15.010266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:15.010306 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:41:15.010949 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:15.010954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:15.010997 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:15.011122 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:15.011862 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.012732 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.014197 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.014587 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.014667 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:15.017188 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:15.017192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:15.017192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:15.017201 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:15.018830 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:15.019253 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:15.019256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:15.019256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:15.019347 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:15.019607 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:15.019608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:15.019608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:15.019867 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:15.019961 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:15.020237 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 02:41:15.020738 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260928 02:41:15.021662 22667 heartbeater.cc:347] Connected to a master server at 127.9.130.62:45265
I20260928 02:41:15.021754 22667 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:15.021914 22667 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
W20260928 02:41:15.022661 22667 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:45265 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:41:15.022738 22667 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:15.022801 22667 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
I20260928 02:41:15.023121 22667 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:15.023243 22667 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
W20260928 02:41:15.023522 22667 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:41:15.134642 22671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:15.134970 22671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:15.135056 22671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:15.136772 22671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:15.136833 22671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:15.136874 22671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:41:15.139062 22671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.130.20:41959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:35253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=38839
--tserver_master_addrs=127.9.130.62:45265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:15.139292 22671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:15.139554 22671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:15.145531 22677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:15.145819 22679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:15.145556 22676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:15.147792 22671 server_base.cc:1034] running on GCE node
I20260928 02:41:15.148043 22671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:15.148283 22671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:15.149463 22671 hybrid_clock.cc:648] HybridClock initialized: now 1790563275149438 us; error 43 us; skew 500 ppm
Sep 28 02:41:15 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563275, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:15.153318 22671 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:41:15.153800 22671 webserver.cc:460] Webserver started at http://127.9.130.2:38839/ using document root <none> and password file <none>
I20260928 02:41:15.154069 22671 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:15.154201 22671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:41:15 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563275, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:41:15.173532 22671 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:15.176263 22687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:15.180032 22671 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260928 02:41:15.180204 22671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "5dac3c1d0f66457c9eed52763d160c5e"
format_stamp: "Formatted at 2026-09-28 02:41:14 on dist-test-slave-kkrh"
server_key: "ed8802055140863aec1ec286a6d978e2"
server_key_iv: "4c1b151a7095e491bf9bc529883d43cc"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:15.180747 22671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:15.210939 22671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:15.213430 22671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:15.213603 22671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:15.213909 22671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:15.214298 22671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:15.214355 22671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:15.214390 22671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:15.214423 22671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:15.221809 22671 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:35253
I20260928 02:41:15.221942 22800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:35253 every 8 connection(s)
I20260928 02:41:15.222358 22671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:15.223842 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 22671
I20260928 02:41:15.224133 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.130.20:41959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:43677
--local_ip_for_outbound_sockets=127.9.130.3
--tserver_master_addrs=127.9.130.62:45265
--webserver_port=40207
--webserver_interface=127.9.130.3
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.130.20:41959/kms with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:41:15 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563275, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:15.233961 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:15.223316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:53137 (local address 127.9.130.62:45265)
0928 02:41:15.223375 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:15.223377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:15.223609 (+ 232us) server_negotiation.cc:407] Connection header received
0928 02:41:15.224349 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:15.224350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:15.224366 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:15.224425 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:15.224982 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.225603 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.227087 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.227310 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.227387 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:15.230482 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:15.230484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:15.230484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:15.230492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:15.232253 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:15.232749 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:15.232750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:15.232751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:15.232822 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:15.233112 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:15.233113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:15.233114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:15.233247 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:15.233292 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:15.233761 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 02:41:15.233865 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:41:15.235011 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:15.223297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:45265 (local address 127.9.130.2:53137)
0928 02:41:15.223489 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:15.223525 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:15.224228 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:15.224497 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:15.224501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:15.224690 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:15.224882 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:15.224886 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.225715 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.225717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:15.226993 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:15.226997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.228023 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.228025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:15.228164 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:15.228735 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:15.228745 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:15.230364 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:15.232354 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:15.232359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:15.232361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:15.232656 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:15.232895 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:15.232896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:15.232896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:15.233034 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:15.233343 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:15.233349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:15.233592 (+ 243us) client_negotiation.cc:769] Sending connection context
0928 02:41:15.234298 (+ 706us) client_negotiation.cc:240] Negotiation successful
0928 02:41:15.234455 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
I20260928 02:41:15.235718 22801 heartbeater.cc:347] Connected to a master server at 127.9.130.62:45265
I20260928 02:41:15.235867 22801 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:15.236053 22801 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
W20260928 02:41:15.236689 22801 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:45265 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:41:15.236778 22801 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:15.236845 22801 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
I20260928 02:41:15.237114 22801 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:15.237187 22801 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
W20260928 02:41:15.237511 22801 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:41:15.344445 22804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:15.344846 22804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:15.344941 22804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:15.346634 22804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:15.346695 22804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:15.346743 22804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:41:15.348384 22804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.130.20:41959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:43677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=40207
--tserver_master_addrs=127.9.130.62:45265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:15.348609 22804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:15.348863 22804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:15.355001 22810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:15.354986 22811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:15.355211 22804 server_base.cc:1034] running on GCE node
W20260928 02:41:15.355320 22813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:15.355526 22804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:15.355774 22804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:15.356988 22804 hybrid_clock.cc:648] HybridClock initialized: now 1790563275356970 us; error 34 us; skew 500 ppm
Sep 28 02:41:15 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563275, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:15.360064 22804 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:41:15.360565 22804 webserver.cc:460] Webserver started at http://127.9.130.3:40207/ using document root <none> and password file <none>
I20260928 02:41:15.360802 22804 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:15.360885 22804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:41:15 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563275, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:41:15.379586 22804 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:15.380718 22821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:15.380956 22804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:41:15.381070 22804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9"
format_stamp: "Formatted at 2026-09-28 02:41:14 on dist-test-slave-kkrh"
server_key: "5c26d883a4035d27db5433b76378c05f"
server_key_iv: "e0668b056c5250e59af478069d3f99b7"
server_key_version: "kuduclusterkey@0"
I20260928 02:41:15.381553 22804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:15.394578 22804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:15.396521 22804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:15.396653 22804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:15.396884 22804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:15.397190 22804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:15.397229 22804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:15.397257 22804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:15.397275 22804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:15.404528 22804 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:43677
I20260928 02:41:15.404628 22934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:43677 every 8 connection(s)
I20260928 02:41:15.405062 22804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:15.406610 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 22804
Sep 28 02:41:15 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563275, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:15.416357 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:15.405801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:39259 (local address 127.9.130.62:45265)
0928 02:41:15.405860 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:15.405863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:15.406110 (+ 247us) server_negotiation.cc:407] Connection header received
0928 02:41:15.406796 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:15.406798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:15.406811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:15.406858 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:15.407686 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.408445 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.409948 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.410232 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.410274 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:15.412916 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:15.412918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:15.412919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:15.412927 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:15.414636 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:15.415165 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:15.415167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:15.415167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:15.415227 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:15.415519 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:15.415521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:15.415521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:15.415679 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:15.415731 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:15.416183 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 02:41:15.416262 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:41:15.417060 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:15.405789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:45265 (local address 127.9.130.3:39259)
0928 02:41:15.405976 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:15.406022 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:15.406662 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:15.407045 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:15.407049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:15.407290 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:15.407574 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:15.407581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.408565 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.408566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:15.409821 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:15.409846 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.410386 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.410387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:15.410521 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:15.411097 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:15.411107 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:15.412771 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:15.414758 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:15.414763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:15.414766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:15.415069 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:15.415299 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:15.415299 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:15.415300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:15.415430 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:15.415785 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:15.415811 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:15.416005 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 02:41:15.416707 (+ 702us) client_negotiation.cc:240] Negotiation successful
0928 02:41:15.416886 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260928 02:41:15.417650 22935 heartbeater.cc:347] Connected to a master server at 127.9.130.62:45265
I20260928 02:41:15.417779 22935 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:15.417959 22935 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
W20260928 02:41:15.418651 22935 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:45265 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:41:15.418736 22935 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:15.418828 22935 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
I20260928 02:41:15.419131 22935 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:15.419203 22935 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
W20260928 02:41:15.419466 22935 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:41:15.421967 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:15.407368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.9.130.62:45265)
0928 02:41:15.407667 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:15.407669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:15.407677 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:41:15.407699 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:15.407700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:15.407711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:15.407801 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:15.408723 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.409920 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.412808 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:15.413122 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:15.413473 (+ 351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:15.417122 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:15.417124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:15.417124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:15.417132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:15.420798 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:15.421145 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:15.421146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:15.421146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:15.421209 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:15.421407 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:15.421408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:15.421408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:15.421540 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:15.421701 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:15.421784 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:41:15.421872 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":63,"threads_started":1}
Sep 28 02:41:15 dist-test-slave-kkrh krb5kdc[21473](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563275, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:16.023728 22667 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:16.023900 22667 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
I20260928 02:41:16.024278 22469 ts_manager.cc:194] Registered new tserver with Master: 6c19c57215d947f8bd7464eaa2daa084 (127.9.130.1:41953)
I20260928 02:41:16.025311 22469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:49899
I20260928 02:41:16.237708 22801 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:16.237891 22801 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
I20260928 02:41:16.238214 22469 ts_manager.cc:194] Registered new tserver with Master: 5dac3c1d0f66457c9eed52763d160c5e (127.9.130.2:35253)
I20260928 02:41:16.238761 22469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:53137
Sep 28 02:41:16 dist-test-slave-kkrh krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563275, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for HTTP/127.9.130.20@KRBTEST.COM
I20260928 02:41:16.419621 22935 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:16.419777 22935 heartbeater.cc:511] Master 127.9.130.62:45265 requested a full tablet report, sending...
I20260928 02:41:16.420111 22468 ts_manager.cc:194] Registered new tserver with Master: f8d2ab9f9a4d41d490dfbd9d17168ad9 (127.9.130.3:43677)
I20260928 02:41:16.420547 22468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:39259
I20260928 02:41:16.420814 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:41:16.428136 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:16.421749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39230 (local address 127.9.130.62:45265)
0928 02:41:16.421861 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:16.421863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:16.421904 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:41:16.421989 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:16.421990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:16.422000 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:16.422096 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:16.422678 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.423175 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.423926 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.424132 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.424233 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:16.425441 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:16.425443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:16.425443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:16.425450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:16.426962 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:16.427324 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:16.427325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:16.427326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:16.427384 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:16.427580 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:16.427581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:16.427581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:16.427689 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:16.427804 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:16.427927 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 02:41:16.428008 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
I20260928 02:41:16.430047 22468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:16.431682 22468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:16.439491 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:16.434288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:43677 (local address 127.0.0.1:41314)
0928 02:41:16.435946 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:16.435977 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:16.436058 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:16.436253 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:16.436256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:16.436258 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:16.436322 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:16.436323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.437326 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.437327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:16.438220 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:16.438223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.439183 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.439184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:16.439271 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:16.439286 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 02:41:16.439352 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:41:16.439403 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1623,"thread_start_us":85,"threads_started":1}
I20260928 02:41:16.439796 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:16.434530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.9.130.3:43677)
0928 02:41:16.434662 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:16.434667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:16.436047 (+ 1380us) server_negotiation.cc:407] Connection header received
0928 02:41:16.436145 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:16.436148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:16.436162 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:16.436252 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:16.436380 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.437233 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.438347 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.438983 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.439079 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:16.439141 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:16.439666 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 02:41:16.439709 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":50,"threads_started":1}
I20260928 02:41:16.440471 22869 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc7e08016724b42ae0106046dd65468 (DEFAULT_TABLE table=test-table [id=175fb49d78b34776b842f436b7f612eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:16.440848 22869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc7e08016724b42ae0106046dd65468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:16.444360 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:16.434340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:41953 (local address 127.0.0.1:43506)
0928 02:41:16.440189 (+ 5849us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:16.440197 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:16.440233 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:16.440513 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:16.440514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:16.440516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:16.440561 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:16.440563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.441747 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.441748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:16.443081 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:16.443084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.444116 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.444117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:16.444185 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:16.444190 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:41:16.444240 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:41:16.444277 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5836,"thread_start_us":99,"threads_started":1}
I20260928 02:41:16.444725 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:16.434665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.9.130.1:41953)
0928 02:41:16.434775 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:16.434778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:16.440300 (+ 5522us) server_negotiation.cc:407] Connection header received
0928 02:41:16.440341 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:16.440343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:16.440358 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:16.440426 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:16.440637 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.441651 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.443179 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.443958 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.444023 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:16.444081 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:16.444595 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 02:41:16.444646 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":54,"threads_started":1}
I20260928 02:41:16.445559 22597 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc7e08016724b42ae0106046dd65468 (DEFAULT_TABLE table=test-table [id=175fb49d78b34776b842f436b7f612eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:16.445847 22597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc7e08016724b42ae0106046dd65468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:16.448450 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:16.434517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.9.130.2:35253)
0928 02:41:16.434662 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:16.434667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:16.443943 (+ 9276us) server_negotiation.cc:407] Connection header received
0928 02:41:16.443986 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:16.443989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:16.444007 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:16.444080 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:16.444969 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.445990 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.447261 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.447998 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.448049 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:16.448102 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:16.448330 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 02:41:16.448370 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260928 02:41:16.448650 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:16.434316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:35253 (local address 127.0.0.1:59650)
0928 02:41:16.440051 (+ 5735us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:16.440061 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:16.440116 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:16.444833 (+ 4717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:16.444834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:16.444836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:16.444871 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:16.444872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.446088 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.446089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:16.447124 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:16.447127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.448175 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.448175 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:16.448234 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:16.448238 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:41:16.448529 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 02:41:16.448576 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5712,"thread_start_us":73,"threads_started":1}
I20260928 02:41:16.449314 22735 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc7e08016724b42ae0106046dd65468 (DEFAULT_TABLE table=test-table [id=175fb49d78b34776b842f436b7f612eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:16.449633 22735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc7e08016724b42ae0106046dd65468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:16.449718 22969 tablet_bootstrap.cc:492] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084: Bootstrap starting.
I20260928 02:41:16.450258 22969 tablet_bootstrap.cc:654] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:16.450469 22969 log.cc:824] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:16.451099 22969 tablet_bootstrap.cc:492] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084: No bootstrap required, opened a new log
I20260928 02:41:16.451143 22969 ts_tablet_manager.cc:1397] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:16.451992 22968 tablet_bootstrap.cc:492] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9: Bootstrap starting.
I20260928 02:41:16.452687 22968 tablet_bootstrap.cc:654] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:16.452965 22968 log.cc:824] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:16.453639 22972 tablet_bootstrap.cc:492] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: Bootstrap starting.
I20260928 02:41:16.454042 22969 raft_consensus.cc:348] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43677 } } peers { permanent_uuid: "6c19c57215d947f8bd7464eaa2daa084" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 41953 } } peers { permanent_uuid: "5dac3c1d0f66457c9eed52763d160c5e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35253 } }
I20260928 02:41:16.454258 22969 raft_consensus.cc:374] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:16.454228 22972 tablet_bootstrap.cc:654] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:16.454327 22969 raft_consensus.cc:729] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c19c57215d947f8bd7464eaa2daa084, State: Initialized, Role: FOLLOWER
I20260928 02:41:16.454488 22969 consensus_queue.cc:260] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43677 } } peers { permanent_uuid: "6c19c57215d947f8bd7464eaa2daa084" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 41953 } } peers { permanent_uuid: "5dac3c1d0f66457c9eed52763d160c5e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35253 } }
I20260928 02:41:16.454737 22972 log.cc:824] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:16.454772 22969 ts_tablet_manager.cc:1428] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 02:41:16.454946 22667 heartbeater.cc:503] Master 127.9.130.62:45265 was elected leader, sending a full tablet report...
I20260928 02:41:16.455513 22968 tablet_bootstrap.cc:492] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9: No bootstrap required, opened a new log
I20260928 02:41:16.455624 22968 ts_tablet_manager.cc:1397] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:41:16.458030 22968 raft_consensus.cc:348] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43677 } } peers { permanent_uuid: "6c19c57215d947f8bd7464eaa2daa084" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 41953 } } peers { permanent_uuid: "5dac3c1d0f66457c9eed52763d160c5e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35253 } }
I20260928 02:41:16.458194 22968 raft_consensus.cc:374] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:16.458223 22968 raft_consensus.cc:729] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d2ab9f9a4d41d490dfbd9d17168ad9, State: Initialized, Role: FOLLOWER
I20260928 02:41:16.458374 22968 consensus_queue.cc:260] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43677 } } peers { permanent_uuid: "6c19c57215d947f8bd7464eaa2daa084" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 41953 } } peers { permanent_uuid: "5dac3c1d0f66457c9eed52763d160c5e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35253 } }
I20260928 02:41:16.458658 22968 ts_tablet_manager.cc:1428] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:16.458770 22935 heartbeater.cc:503] Master 127.9.130.62:45265 was elected leader, sending a full tablet report...
I20260928 02:41:16.459470 22972 tablet_bootstrap.cc:492] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: No bootstrap required, opened a new log
I20260928 02:41:16.459535 22972 ts_tablet_manager.cc:1397] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 02:41:16.461884 22972 raft_consensus.cc:348] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43677 } } peers { permanent_uuid: "6c19c57215d947f8bd7464eaa2daa084" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 41953 } } peers { permanent_uuid: "5dac3c1d0f66457c9eed52763d160c5e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35253 } }
I20260928 02:41:16.462031 22972 raft_consensus.cc:374] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:16.462061 22972 raft_consensus.cc:729] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dac3c1d0f66457c9eed52763d160c5e, State: Initialized, Role: FOLLOWER
I20260928 02:41:16.462189 22972 consensus_queue.cc:260] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43677 } } peers { permanent_uuid: "6c19c57215d947f8bd7464eaa2daa084" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 41953 } } peers { permanent_uuid: "5dac3c1d0f66457c9eed52763d160c5e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35253 } }
I20260928 02:41:16.462414 22972 ts_tablet_manager.cc:1428] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:16.462513 22801 heartbeater.cc:503] Master 127.9.130.62:45265 was elected leader, sending a full tablet report...
W20260928 02:41:16.478731 22802 tablet.cc:2369] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:16.499974 22973 raft_consensus.cc:482] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:16.500229 22973 raft_consensus.cc:504] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43677 } } peers { permanent_uuid: "6c19c57215d947f8bd7464eaa2daa084" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 41953 } } peers { permanent_uuid: "5dac3c1d0f66457c9eed52763d160c5e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35253 } }
I20260928 02:41:16.501431 22973 leader_election.cc:288] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8d2ab9f9a4d41d490dfbd9d17168ad9 (127.9.130.3:43677), 5dac3c1d0f66457c9eed52763d160c5e (127.9.130.2:35253)
I20260928 02:41:16.504612 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:16.501155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:33105 (local address 127.9.130.3:43677)
0928 02:41:16.501271 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:16.501273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:16.501282 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:41:16.501310 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:16.501311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:16.501322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:16.501389 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:16.501566 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.502322 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.503671 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.504242 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.504287 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:16.504296 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:16.504510 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 02:41:16.504547 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 02:41:16.504829 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:16.500908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:43677 (local address 127.9.130.1:33105)
0928 02:41:16.501073 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:16.501083 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:16.501129 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:16.501418 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:16.501419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:16.501421 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:16.501493 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:16.501495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.502420 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.502421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:16.503571 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:16.503574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.504369 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.504370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:16.504434 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:16.504437 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:41:16.504693 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 02:41:16.504734 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260928 02:41:16.505119 22889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc7e08016724b42ae0106046dd65468" candidate_uuid: "6c19c57215d947f8bd7464eaa2daa084" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9" is_pre_election: true
I20260928 02:41:16.505297 22889 raft_consensus.cc:2454] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c19c57215d947f8bd7464eaa2daa084 in term 0.
I20260928 02:41:16.505553 22557 leader_election.cc:302] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c19c57215d947f8bd7464eaa2daa084, f8d2ab9f9a4d41d490dfbd9d17168ad9; no voters:
I20260928 02:41:16.505692 22973 raft_consensus.cc:2781] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:16.505744 22973 raft_consensus.cc:482] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:16.505784 22973 raft_consensus.cc:3037] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:16.506736 22973 raft_consensus.cc:504] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43677 } } peers { permanent_uuid: "6c19c57215d947f8bd7464eaa2daa084" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 41953 } } peers { permanent_uuid: "5dac3c1d0f66457c9eed52763d160c5e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35253 } }
I20260928 02:41:16.506989 22973 leader_election.cc:288] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [CANDIDATE]: Term 1 election: Requested vote from peers f8d2ab9f9a4d41d490dfbd9d17168ad9 (127.9.130.3:43677), 5dac3c1d0f66457c9eed52763d160c5e (127.9.130.2:35253)
I20260928 02:41:16.507036 22889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc7e08016724b42ae0106046dd65468" candidate_uuid: "6c19c57215d947f8bd7464eaa2daa084" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9"
I20260928 02:41:16.507110 22889 raft_consensus.cc:3037] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:16.508030 22889 raft_consensus.cc:2454] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c19c57215d947f8bd7464eaa2daa084 in term 1.
I20260928 02:41:16.508044 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:16.501215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:38757 (local address 127.9.130.2:35253)
0928 02:41:16.501276 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:16.501278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:16.502537 (+ 1259us) server_negotiation.cc:407] Connection header received
0928 02:41:16.502633 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:16.502634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:16.502646 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:16.502674 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:16.502912 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.503576 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.507081 (+ 3505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.507610 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.507660 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:16.507671 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:16.507931 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 02:41:16.507976 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:41:16.508188 22557 leader_election.cc:302] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c19c57215d947f8bd7464eaa2daa084, f8d2ab9f9a4d41d490dfbd9d17168ad9; no voters:
I20260928 02:41:16.508278 22973 raft_consensus.cc:2781] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:16.508426 22973 raft_consensus.cc:686] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [term 1 LEADER]: Becoming Leader. State: Replica: 6c19c57215d947f8bd7464eaa2daa084, State: Running, Role: LEADER
I20260928 02:41:16.508528 22973 consensus_queue.cc:237] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43677 } } peers { permanent_uuid: "6c19c57215d947f8bd7464eaa2daa084" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 41953 } } peers { permanent_uuid: "5dac3c1d0f66457c9eed52763d160c5e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35253 } }
I20260928 02:41:16.509039 22978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:16.500909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:35253 (local address 127.9.130.1:38757)
0928 02:41:16.502482 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:16.502492 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:16.502562 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:16.502730 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:16.502731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:16.502733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:16.502834 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:16.502835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.503687 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.503687 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:16.505354 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:16.505357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.507732 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.507733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:16.507841 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:16.507846 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:41:16.508157 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 02:41:16.508197 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1555,"mutex_wait_us":30,"thread_start_us":86,"threads_started":1}
I20260928 02:41:16.509363 22752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc7e08016724b42ae0106046dd65468" candidate_uuid: "6c19c57215d947f8bd7464eaa2daa084" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dac3c1d0f66457c9eed52763d160c5e" is_pre_election: true
I20260928 02:41:16.509509 22752 raft_consensus.cc:2454] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c19c57215d947f8bd7464eaa2daa084 in term 0.
I20260928 02:41:16.509418 22468 catalog_manager.cc:5495] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c19c57215d947f8bd7464eaa2daa084 (127.9.130.1). New cstate: current_term: 1 leader_uuid: "6c19c57215d947f8bd7464eaa2daa084" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c19c57215d947f8bd7464eaa2daa084" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dac3c1d0f66457c9eed52763d160c5e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35253 } health_report { overall_health: UNKNOWN } } }
I20260928 02:41:16.509642 22749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc7e08016724b42ae0106046dd65468" candidate_uuid: "6c19c57215d947f8bd7464eaa2daa084" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dac3c1d0f66457c9eed52763d160c5e"
I20260928 02:41:16.509706 22749 raft_consensus.cc:3037] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:16.510526 22749 raft_consensus.cc:2454] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c19c57215d947f8bd7464eaa2daa084 in term 1.
W20260928 02:41:16.511013 22668 tablet.cc:2369] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:16.519415 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:16.515777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.9.130.1:41953)
0928 02:41:16.515831 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:16.515833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:16.515871 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:41:16.515995 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:16.515995 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:16.516014 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:16.516103 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:16.516584 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.517290 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.518435 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:16.518710 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:16.518802 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:16.518962 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:16.519074 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:16.519152 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:16.519332 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 02:41:16.519353 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:41:16.521899 22749 raft_consensus.cc:1261] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e [term 1 FOLLOWER]: Refusing update from remote peer 6c19c57215d947f8bd7464eaa2daa084: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:16.522174 22889 raft_consensus.cc:1261] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6c19c57215d947f8bd7464eaa2daa084: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:16.522488 22979 consensus_queue.cc:1035] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8d2ab9f9a4d41d490dfbd9d17168ad9" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:16.522645 22973 consensus_queue.cc:1035] T 8bc7e08016724b42ae0106046dd65468 P 6c19c57215d947f8bd7464eaa2daa084 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dac3c1d0f66457c9eed52763d160c5e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:16.527333 22983 mvcc.cc:205] Tried to move back new op lower bound from 7334147180630990848 to 7334147180579667968. Current Snapshot: MvccSnapshot[applied={T|T < 7334147180630990848}]
I20260928 02:41:16.527418 22982 mvcc.cc:205] Tried to move back new op lower bound from 7334147180630990848 to 7334147180579667968. Current Snapshot: MvccSnapshot[applied={T|T < 7334147180630990848 or (T in {7334147180630990848})}]
I20260928 02:41:16.528342 22468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:16.528426 22468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:16.529138 22468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7200948713d46d78b3958c12c62adae: Sending DeleteTablet for 3 replicas of tablet 8bc7e08016724b42ae0106046dd65468
I20260928 02:41:16.529450 22869 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bc7e08016724b42ae0106046dd65468 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:16 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:41314
I20260928 02:41:16.529445 22597 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bc7e08016724b42ae0106046dd65468 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:16 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:43506
I20260928 02:41:16.529541 22735 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bc7e08016724b42ae0106046dd65468 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:16 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:59650
I20260928 02:41:16.529964 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 22507
I20260928 02:41:16.535964 22992 tablet_replica.cc:330] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: stopping tablet replica
I20260928 02:41:16.536092 22992 raft_consensus.cc:2229] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:16.536172 22992 pending_rounds.cc:62] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: Trying to abort 2 pending ops.
I20260928 02:41:16.536213 22992 pending_rounds.cc:68] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334147180630990848 op_type: WRITE_OP write_request { tablet_id: "8bc7e08016724b42ae0106046dd65468" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\370\251\347\325\006\"8\n\ntest-admin\022*\n 175fb49d78b34776b842f436b7f612eb\020\001\030\001 \001(\001" signature: "<redacted>""o\373$E\177pDt\252\356\311>\301\226\243\266\260w\220I^\201rq|\253\023S`n!\2134\306\n@c\206\327\016e3_\025\356\243&\234^\260\362\2644v\311m\211\252\207\372\346\3364\242" signing_key_seq_num: 0 } } request_id { client_id: "fcaae10df46d4cdf9f54c2221741431a" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260928 02:41:16.536432 22992 pending_rounds.cc:68] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334147180579667968 op_type: NO_OP noop_request { }
I20260928 02:41:16.536486 22992 raft_consensus.cc:2866] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260928 02:41:16.536521 22992 raft_consensus.cc:2258] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:16.536665 22990 tablet_replica.cc:330] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9: stopping tablet replica
I20260928 02:41:16.536757 22990 raft_consensus.cc:2229] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:16.536866 22990 raft_consensus.cc:2258] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:16.537122 22992 ts_tablet_manager.cc:1905] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:16.537293 22990 ts_tablet_manager.cc:1905] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:16.539058 22990 ts_tablet_manager.cc:1918] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:16.539058 22992 ts_tablet_manager.cc:1918] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:16.539136 22990 log.cc:1196] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/8bc7e08016724b42ae0106046dd65468
I20260928 02:41:16.539232 22992 log.cc:1196] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/8bc7e08016724b42ae0106046dd65468
I20260928 02:41:16.539449 22990 ts_tablet_manager.cc:1939] T 8bc7e08016724b42ae0106046dd65468 P f8d2ab9f9a4d41d490dfbd9d17168ad9: Deleting consensus metadata
I20260928 02:41:16.539592 22992 ts_tablet_manager.cc:1939] T 8bc7e08016724b42ae0106046dd65468 P 5dac3c1d0f66457c9eed52763d160c5e: Deleting consensus metadata
I20260928 02:41:16.539888 22456 catalog_manager.cc:4841] TS f8d2ab9f9a4d41d490dfbd9d17168ad9 (127.9.130.3:43677): tablet 8bc7e08016724b42ae0106046dd65468 (table test-table [id=175fb49d78b34776b842f436b7f612eb]) successfully deleted
I20260928 02:41:16.540174 22454 catalog_manager.cc:4841] TS 5dac3c1d0f66457c9eed52763d160c5e (127.9.130.2:35253): tablet 8bc7e08016724b42ae0106046dd65468 (table test-table [id=175fb49d78b34776b842f436b7f612eb]) successfully deleted
W20260928 02:41:16.542706 22691 connection.cc:535] server connection from 127.9.130.1:38757 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:41:16.543160 22453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:41953 (error 108)
I20260928 02:41:16.543677 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:16.543534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:41953 (local address 127.0.0.1:43530)
0928 02:41:16.543581 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:16.543635 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:41953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 02:41:16.543807 22453 catalog_manager.cc:4584] TS 6c19c57215d947f8bd7464eaa2daa084 (127.9.130.1:41953): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bc7e08016724b42ae0106046dd65468: Network error: Client connection negotiation failed: client connection to 127.9.130.1:41953: connect: Connection refused (error 111)
I20260928 02:41:16.543862 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 22671
I20260928 02:41:16.548581 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 22804
I20260928 02:41:16.555835 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 22436
I20260928 02:41:16.568217 9736 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:41:16.902371 9736 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:41:16.902458 9736 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:41:17.246366 9736 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:41:17.246 UTC [21503] LOG: received smart shutdown request
2026-09-28 02:41:17.250 UTC [21503] LOG: background worker "logical replication launcher" (PID 21512) exited with exit code 1
2026-09-28 02:41:17.250 UTC [21507] LOG: shutting down
2026-09-28 02:41:17.425 UTC [21503] LOG: database system is shut down
2026-09-28T02:41:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47603 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:17 dist-test-slave-kkrh krb5kdc[23002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:17 dist-test-slave-kkrh krb5kdc[23002](info): set up 2 sockets
Sep 28 02:41:17 dist-test-slave-kkrh krb5kdc[23002](info): commencing operation
krb5kdc: starting...
W20260928 02:41:18.575712 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:41:18 dist-test-slave-kkrh krb5kdc[23002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:41:18.695029 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:35811
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:35811,127.9.130.61:35945,127.9.130.60:37877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:18.776825 23015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:18.777053 23015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:18.777088 23015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:18.778249 23015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:18.778311 23015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:18.778337 23015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:18.778370 23015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:18.778395 23015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:18.779999 23015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:35811,127.9.130.61:35945,127.9.130.60:37877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:35811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:18.780272 23015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:18.780498 23015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:18.785230 23020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:18.785239 23023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:18.785246 23021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:18.785529 23015 server_base.cc:1034] running on GCE node
I20260928 02:41:18.785879 23015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:18.786217 23015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:18.787448 23015 hybrid_clock.cc:648] HybridClock initialized: now 1790563278787436 us; error 41 us; skew 500 ppm
Sep 28 02:41:18 dist-test-slave-kkrh krb5kdc[23002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563278, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:18.793919 23015 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:41:18.794401 23015 webserver.cc:460] Webserver started at http://127.9.130.62:46123/ using document root <none> and password file <none>
I20260928 02:41:18.794698 23015 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:18.794770 23015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:18.794919 23015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:18.796066 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "6d04c18bf19d4230aa59e04682a40963"
format_stamp: "Formatted at 2026-09-28 02:41:18 on dist-test-slave-kkrh"
server_key: "de5aeaf1949cd61c984c2c3c8e38f35c"
server_key_iv: "423a1ebf2f4abcc0f79a4bff07b69899"
server_key_version: "encryptionkey@0"
I20260928 02:41:18.796481 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "6d04c18bf19d4230aa59e04682a40963"
format_stamp: "Formatted at 2026-09-28 02:41:18 on dist-test-slave-kkrh"
server_key: "de5aeaf1949cd61c984c2c3c8e38f35c"
server_key_iv: "423a1ebf2f4abcc0f79a4bff07b69899"
server_key_version: "encryptionkey@0"
I20260928 02:41:18.797853 23015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:18.798626 23030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:18.798861 23015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:18.798949 23015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "6d04c18bf19d4230aa59e04682a40963"
format_stamp: "Formatted at 2026-09-28 02:41:18 on dist-test-slave-kkrh"
server_key: "de5aeaf1949cd61c984c2c3c8e38f35c"
server_key_iv: "423a1ebf2f4abcc0f79a4bff07b69899"
server_key_version: "encryptionkey@0"
I20260928 02:41:18.799042 23015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:18.831501 23015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:18.833532 23015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:18.833683 23015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:18.837967 23015 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:35811
I20260928 02:41:18.838011 23082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:35811 every 8 connection(s)
I20260928 02:41:18.838387 23015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:18.839432 23083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:18.839989 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 23015
I20260928 02:41:18.840073 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:41:18.840266 9736 external_mini_cluster.cc:1428] Setting key f470c0dbbeb6fc36b2660616a412d976
I20260928 02:41:18.841444 23083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.130.61@KRBTEST.COM; defaulting to no policy
Sep 28 02:41:18 dist-test-slave-kkrh krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563278, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.61@KRBTEST.COM" created.
I20260928 02:41:18.852030 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:18.842666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.9.130.62:35811)
0928 02:41:18.842811 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:18.842816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:18.842848 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:41:18.843336 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:18.843338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:18.843359 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:18.843471 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:18.844292 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:18.845279 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:18.846176 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:18.846502 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:18.846629 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:18.848781 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:18.848787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:18.848788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:18.848794 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:18.850375 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:18.850789 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:18.850791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:18.850792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:18.850874 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:18.851120 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:18.851121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:18.851122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:18.851354 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:18.851447 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:18.851627 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 02:41:18.851772 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260928 02:41:18.852029 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:18.842411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:35811 (local address 127.0.0.1:47622)
0928 02:41:18.842613 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:18.842664 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:18.843176 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:18.843482 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:18.843485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:18.843946 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:18.844181 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:18.844188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:18.845389 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:18.845390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:18.846067 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:18.846069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:18.846600 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:18.846601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:18.846674 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:18.847246 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:18.847262 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:18.848684 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:18.850479 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:18.850485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:18.850487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:18.850686 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:18.850941 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:18.850941 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:18.850942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:18.851031 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:18.851428 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:18.851432 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:18.851493 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 02:41:18.851602 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:41:18.851841 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":80,"thread_start_us":87,"threads_started":1}
I20260928 02:41:18.853055 23083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:41:18.853436 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:18.853302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:35945 (local address 127.0.0.1:45808)
0928 02:41:18.853352 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:18.853401 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:41:18.853639 23034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.130.61:35945: connect: Connection refused (error 111)
I20260928 02:41:18.854104 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:18.854013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:35945 (local address 127.0.0.1:45814)
0928 02:41:18.854064 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:18.854087 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 02:41:18.854203 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.61:35945: Network error: Client connection negotiation failed: client connection to 127.9.130.61:35945: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.61.keytab.
Entry for principal kudu/127.9.130.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.61.keytab.
I20260928 02:41:18.867460 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.61:35945
--webserver_interface=127.9.130.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.61.keytab
--principal=kudu/127.9.130.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:35811,127.9.130.61:35945,127.9.130.60:37877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:41:18.903417 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } attempt: 1
I20260928 02:41:18.903905 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:18.903757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:35945 (local address 127.0.0.1:45830)
0928 02:41:18.903836 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:18.903876 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 02:41:18.904228 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:18.904133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:35945 (local address 127.0.0.1:45840)
0928 02:41:18.904178 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:18.904204 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:41:18.904314 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.61:35945: Network error: Client connection negotiation failed: client connection to 127.9.130.61:35945: connect: Connection refused (error 111)
W20260928 02:41:18.947181 23090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:18.947413 23090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:18.947451 23090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:18.948632 23090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:18.948700 23090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:18.948724 23090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:18.948753 23090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:18.948786 23090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:18.950264 23090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:35811,127.9.130.61:35945,127.9.130.60:37877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.61.keytab
--principal=kudu/127.9.130.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.61:35945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.130.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:18.950544 23090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:18.950768 23090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:18.955679 23095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:18.955636 23096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:18.955632 23098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:18.955843 23090 server_base.cc:1034] running on GCE node
I20260928 02:41:18.956146 23090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:18.956377 23090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:18.957523 23090 hybrid_clock.cc:648] HybridClock initialized: now 1790563278957512 us; error 27 us; skew 500 ppm
Sep 28 02:41:18 dist-test-slave-kkrh krb5kdc[23002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563278, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:18.960315 23090 init.cc:377] Logged in from keytab as kudu/127.9.130.61@KRBTEST.COM (short username kudu)
I20260928 02:41:18.960801 23090 webserver.cc:460] Webserver started at http://127.9.130.61:40601/ using document root <none> and password file <none>
I20260928 02:41:18.961061 23090 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:18.961127 23090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:18.961278 23090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:18.962280 23090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/data/instance:
uuid: "c0451a68581249e391c85f33e0ed5d18"
format_stamp: "Formatted at 2026-09-28 02:41:18 on dist-test-slave-kkrh"
server_key: "4c474dc7b0993337a6b0eccf38203768"
server_key_iv: "632f616c4705d5148403739a21b94db4"
server_key_version: "encryptionkey@0"
I20260928 02:41:18.962638 23090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/wal/instance:
uuid: "c0451a68581249e391c85f33e0ed5d18"
format_stamp: "Formatted at 2026-09-28 02:41:18 on dist-test-slave-kkrh"
server_key: "4c474dc7b0993337a6b0eccf38203768"
server_key_iv: "632f616c4705d5148403739a21b94db4"
server_key_version: "encryptionkey@0"
I20260928 02:41:18.963977 23090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:41:18.964764 23105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:18.964928 23090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:18.965034 23090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/wal
uuid: "c0451a68581249e391c85f33e0ed5d18"
format_stamp: "Formatted at 2026-09-28 02:41:18 on dist-test-slave-kkrh"
server_key: "4c474dc7b0993337a6b0eccf38203768"
server_key_iv: "632f616c4705d5148403739a21b94db4"
server_key_version: "encryptionkey@0"
I20260928 02:41:18.965121 23090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:18.972440 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } attempt: 2
I20260928 02:41:18.972819 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:18.972701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:35945 (local address 127.0.0.1:45850)
0928 02:41:18.972757 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:18.972792 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 02:41:18.973155 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:18.973059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:35945 (local address 127.0.0.1:45866)
0928 02:41:18.973104 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:18.973130 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:41:18.973243 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.61:35945: Network error: Client connection negotiation failed: client connection to 127.9.130.61:35945: connect: Connection refused (error 111)
I20260928 02:41:18.977895 23090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:18.980177 23090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:18.980330 23090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:18.984586 23090 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.61:35945
I20260928 02:41:18.984616 23157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.61:35945 every 8 connection(s)
I20260928 02:41:18.984947 23090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/data/info.pb
I20260928 02:41:18.985760 23158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:18.987474 23158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:41:18.992170 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 23090
I20260928 02:41:18.992282 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-1/wal/instance
I20260928 02:41:18.992496 9736 external_mini_cluster.cc:1428] Setting key 666d67ed9ab3191d8c9ac6e5120a1d42
Sep 28 02:41:18 dist-test-slave-kkrh krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563278, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.61@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:18.996591 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:18.988205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47634 (local address 127.9.130.62:35811)
0928 02:41:18.988267 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:18.988269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:18.988495 (+ 226us) server_negotiation.cc:407] Connection header received
0928 02:41:18.988920 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:18.988921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:18.988931 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:18.988994 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:18.989713 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:18.990201 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:18.991219 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:18.991416 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:18.991499 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:18.993775 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:18.993776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:18.993776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:18.993783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:18.995178 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:18.995598 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:18.995599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:18.995599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:18.995658 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:18.995958 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:18.995959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:18.995959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:18.996076 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:18.996157 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:18.996412 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 02:41:18.996492 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:41:18.996763 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:18.988189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:35811 (local address 127.0.0.1:47634)
0928 02:41:18.988337 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:18.988376 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:18.988840 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:18.989101 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:18.989105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:18.989355 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:18.989614 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:18.989620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:18.990289 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:18.990289 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:18.991134 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:18.991136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:18.991520 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:18.991521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:18.991629 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:18.992133 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:18.992144 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:18.993641 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:18.995279 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:18.995285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:18.995287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:18.995504 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:18.995726 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:18.995727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:18.995727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:18.995880 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:18.996148 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:18.996154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:18.996287 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 02:41:18.996392 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:41:18.996541 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260928 02:41:18.997587 23158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.130.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.60@KRBTEST.COM" created.
Sep 28 02:41:18 dist-test-slave-kkrh krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563278, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.61@KRBTEST.COM for kudu/127.9.130.61@KRBTEST.COM
I20260928 02:41:19.006557 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:18.997840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:35945 (local address 127.0.0.1:45874)
0928 02:41:18.997902 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:18.997916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:18.997966 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:18.998159 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:18.998160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:18.998317 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:18.998374 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:18.998376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:18.999203 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:18.999204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.000065 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:19.000068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.000551 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.000553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.000635 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.000975 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:19.000979 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:19.003721 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:19.005518 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.005519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.005519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.005687 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.005912 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.005912 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.005913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.005970 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.006244 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:19.006244 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:19.006280 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:41:19.006344 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:41:19.006440 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:41:19.006716 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:18.997876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.9.130.61:35945)
0928 02:41:18.998020 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:18.998028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:18.998043 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:41:18.998061 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:18.998062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:18.998072 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:18.998130 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:18.998453 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:18.999111 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.000148 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.000471 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.000526 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.003835 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.003843 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.003844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.003850 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.005415 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.005772 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.005775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.005776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.005843 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.006041 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.006042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.006042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.006174 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.006263 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.006369 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.006506 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260928 02:41:19.007225 23158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:41:19.007561 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.007440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:37877 (local address 127.0.0.1:38092)
0928 02:41:19.007485 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.007533 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:41:19.007678 23107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
I20260928 02:41:19.008101 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.008008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:37877 (local address 127.0.0.1:38102)
0928 02:41:19.008056 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.008080 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:41:19.008201 23158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:37877: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.60.keytab.
Entry for principal kudu/127.9.130.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.60.keytab.
I20260928 02:41:19.018312 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.60:37877
--webserver_interface=127.9.130.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.60.keytab
--principal=kudu/127.9.130.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:35811,127.9.130.61:35945,127.9.130.60:37877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:41:19.057446 23158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } attempt: 1
I20260928 02:41:19.058012 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.057827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:37877 (local address 127.0.0.1:38114)
0928 02:41:19.057898 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.057949 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:41:19.058406 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.058314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:37877 (local address 127.0.0.1:38128)
0928 02:41:19.058359 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.058387 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:41:19.058558 23158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:37877: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
I20260928 02:41:19.064385 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } attempt: 3
Sep 28 02:41:19 dist-test-slave-kkrh krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563278, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for kudu/127.9.130.61@KRBTEST.COM
I20260928 02:41:19.071445 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.064755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.9.130.61:35945)
0928 02:41:19.064817 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:19.064819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:19.064829 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:41:19.064870 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:19.064871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:19.064880 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:19.064950 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:19.065281 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.065784 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.066473 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.066693 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.066780 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.068990 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.068991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.068991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.068997 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.070388 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.070769 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.070769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.070770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.070817 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.071015 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.071016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.071016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.071119 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.071166 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.071328 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.071386 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:41:19.071499 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.064696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:35945 (local address 127.0.0.1:45884)
0928 02:41:19.064738 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.064751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:19.064820 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:19.064953 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:19.064954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:19.065120 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:19.065188 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:19.065190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.065865 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.065866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.066386 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:19.066388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.066779 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.066780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.066859 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.067322 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:19.067328 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:19.068883 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:19.070474 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.070475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.070476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.070687 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.070877 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.070877 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.070878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.070943 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.071180 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:19.071181 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:19.071232 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 02:41:19.071306 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:41:19.071404 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 02:41:19.071913 23083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:41:19.072263 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.072153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:37877 (local address 127.0.0.1:38134)
0928 02:41:19.072202 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.072238 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 02:41:19.072569 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.072479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:37877 (local address 127.0.0.1:38150)
0928 02:41:19.072525 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.072552 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:41:19.072669 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:37877: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
W20260928 02:41:19.100086 23165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:19.100265 23165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:19.100286 23165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:19.101395 23165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:19.101433 23165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:19.101454 23165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:19.101466 23165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:19.101502 23165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:19.103048 23165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:35811,127.9.130.61:35945,127.9.130.60:37877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.60.keytab
--principal=kudu/127.9.130.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.60:37877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.130.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:19.103271 23165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:19.103507 23165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:19.103837 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } attempt: 1
I20260928 02:41:19.104269 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.104113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:37877 (local address 127.0.0.1:38156)
0928 02:41:19.104179 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.104225 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:41:19.104673 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.104565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:37877 (local address 127.0.0.1:38160)
0928 02:41:19.104618 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.104650 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:41:19.104784 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:37877: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
W20260928 02:41:19.108268 23171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:19.108305 23170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:19.108434 23173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:19.108659 23165 server_base.cc:1034] running on GCE node
I20260928 02:41:19.108932 23165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:19.109215 23165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:19.110383 23165 hybrid_clock.cc:648] HybridClock initialized: now 1790563279110349 us; error 45 us; skew 500 ppm
Sep 28 02:41:19 dist-test-slave-kkrh krb5kdc[23002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563279, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:19.112900 23165 init.cc:377] Logged in from keytab as kudu/127.9.130.60@KRBTEST.COM (short username kudu)
I20260928 02:41:19.113301 23165 webserver.cc:460] Webserver started at http://127.9.130.60:38745/ using document root <none> and password file <none>
I20260928 02:41:19.113543 23165 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:19.113610 23165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:19.113744 23165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:19.114804 23165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/data/instance:
uuid: "c7cc90d835a842dba572b12d500a6db3"
format_stamp: "Formatted at 2026-09-28 02:41:19 on dist-test-slave-kkrh"
server_key: "68965628b2791f1290db7ea424881afb"
server_key_iv: "36cc22adb74edd11adcc1a0904f32c38"
server_key_version: "encryptionkey@0"
I20260928 02:41:19.115152 23165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/wal/instance:
uuid: "c7cc90d835a842dba572b12d500a6db3"
format_stamp: "Formatted at 2026-09-28 02:41:19 on dist-test-slave-kkrh"
server_key: "68965628b2791f1290db7ea424881afb"
server_key_iv: "36cc22adb74edd11adcc1a0904f32c38"
server_key_version: "encryptionkey@0"
I20260928 02:41:19.116564 23165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:41:19.117300 23180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:19.117550 23165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:41:19.117636 23165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/wal
uuid: "c7cc90d835a842dba572b12d500a6db3"
format_stamp: "Formatted at 2026-09-28 02:41:19 on dist-test-slave-kkrh"
server_key: "68965628b2791f1290db7ea424881afb"
server_key_iv: "36cc22adb74edd11adcc1a0904f32c38"
server_key_version: "encryptionkey@0"
I20260928 02:41:19.117688 23165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:19.126727 23158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } attempt: 2
I20260928 02:41:19.127104 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.126967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:37877 (local address 127.0.0.1:38176)
0928 02:41:19.127031 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.127069 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:41:19.127457 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.127364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:37877 (local address 127.0.0.1:38182)
0928 02:41:19.127411 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.127437 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 02:41:19.127578 23158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:37877: Network error: Client connection negotiation failed: client connection to 127.9.130.60:37877: connect: Connection refused (error 111)
I20260928 02:41:19.130064 23165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:19.131979 23165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:19.132128 23165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:19.135720 23165 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.60:37877
I20260928 02:41:19.135771 23232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.60:37877 every 8 connection(s)
I20260928 02:41:19.136124 23165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/data/info.pb
I20260928 02:41:19.136968 23233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:19.138394 23233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:41:19.142930 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 23165
I20260928 02:41:19.143054 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563127283700-9736-0/minicluster-data/master-2/wal/instance
I20260928 02:41:19.143246 9736 external_mini_cluster.cc:1428] Setting key 42bc7c0298533538baf1548e0ea230d1
Sep 28 02:41:19 dist-test-slave-kkrh krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563279, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.60@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:19.148527 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.139064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:35811 (local address 127.0.0.1:47636)
0928 02:41:19.139199 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.139229 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:19.139669 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:19.139936 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:19.139939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:19.140171 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:19.140346 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:19.140350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.141033 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.141034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.141800 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:19.141802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.142212 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.142213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.142335 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.142796 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:19.142804 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:19.144386 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:19.147075 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.147080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.147081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.147370 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.147622 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.147622 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.147623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.147757 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.148083 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:19.148088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:19.148192 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 02:41:19.148273 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:41:19.148376 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260928 02:41:19.148550 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.139077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.9.130.62:35811)
0928 02:41:19.139139 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:19.139142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:19.139301 (+ 159us) server_negotiation.cc:407] Connection header received
0928 02:41:19.139771 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:19.139773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:19.139784 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:19.139866 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:19.140438 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.140918 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.141903 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.142102 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.142208 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.144483 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.144485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.144486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.144495 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.147008 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.147445 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.147446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.147446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.147530 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.147846 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.147847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.147848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.148001 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.148094 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.148334 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.148427 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:41:19.149137 23233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:41:19 dist-test-slave-kkrh krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.60@KRBTEST.COM
I20260928 02:41:19.153398 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.143638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.9.130.60:37877)
0928 02:41:19.143830 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:19.143836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:19.143918 (+ 82us) server_negotiation.cc:407] Connection header received
0928 02:41:19.143976 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:19.143978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:19.143999 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:19.144079 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:19.144891 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.145768 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.146889 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.147267 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.147340 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.150192 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.150199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.150199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.150205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.151979 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.152548 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.152550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.152551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.152626 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.152827 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.152828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.152828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.152987 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.153150 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.153237 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.153302 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
Sep 28 02:41:19 dist-test-slave-kkrh krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563279, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.60@KRBTEST.COM for kudu/127.9.130.61@KRBTEST.COM
I20260928 02:41:19.156661 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.149361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:35945 (local address 127.0.0.1:45896)
0928 02:41:19.149504 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.149515 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:19.149570 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:19.149698 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:19.149699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:19.149859 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:19.149905 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:19.149907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.150744 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.150745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.151563 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:19.151567 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.151962 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.151964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.152045 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.152561 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:19.152565 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:19.154013 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:19.155700 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.155702 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.155702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.155926 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.156115 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.156115 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.156116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.156179 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.156376 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:19.156377 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:19.156416 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:41:19.156480 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:41:19.156558 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260928 02:41:19.156669 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.149446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.9.130.61:35945)
0928 02:41:19.149522 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:19.149524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:19.149555 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:41:19.149638 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:19.149639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:19.149646 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:19.149690 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:19.149983 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.150655 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.151652 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.151888 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.151960 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.154073 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.154074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.154074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.154081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.155603 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.156004 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.156005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.156005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.156054 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.156238 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.156239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.156239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.156322 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.156364 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.156504 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.156560 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:41:19.157053 23233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:41:19 dist-test-slave-kkrh krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
Sep 28 02:41:19 dist-test-slave-kkrh krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563279, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.60@KRBTEST.COM for kudu/127.9.130.60@KRBTEST.COM
I20260928 02:41:19.162127 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.154163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.9.130.62:35811)
0928 02:41:19.154231 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:19.154233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:19.154241 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:41:19.154261 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:19.154263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:19.154272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:19.154336 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:19.154906 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.155649 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.156626 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.156900 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.156974 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.159057 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.159058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.159058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.159066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.160853 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.161239 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.161241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.161241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.161296 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.161557 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.161559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.161559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.161686 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.161886 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.161923 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.162017 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:41:19.163276 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.157284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:37877 (local address 127.0.0.1:38204)
0928 02:41:19.157368 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.157378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:19.157420 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:19.157533 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:19.157534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:19.157659 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:19.157681 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:19.157682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.158268 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.158269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.158734 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:19.158736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.159091 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.159092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.159135 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.159447 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:19.159451 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:19.160456 (+ 1005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:19.162316 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.162317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.162317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.162485 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.162728 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.162729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.162730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.162795 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.163050 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:19.163051 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:19.163090 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:41:19.163160 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:41:19.163223 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 02:41:19.163475 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.157337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.9.130.60:37877)
0928 02:41:19.157367 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:19.157368 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:19.157407 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:41:19.157468 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:19.157469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:19.157478 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:19.157530 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:19.157750 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.158184 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.158804 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.159003 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.159090 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.160536 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.160537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.160537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.160545 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.162236 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.162595 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.162596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.162596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.162657 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.162871 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.162872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.162872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.162982 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.163046 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.163191 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.163281 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":14}
I20260928 02:41:19.164872 23233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3: Bootstrap starting.
I20260928 02:41:19.165962 23233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:19.166293 23233 log.cc:824] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:19.167070 23233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3: No bootstrap required, opened a new log
Sep 28 02:41:19 dist-test-slave-kkrh krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.61@KRBTEST.COM
I20260928 02:41:19.169023 23233 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } }
I20260928 02:41:19.169137 23233 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:19.169159 23233 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7cc90d835a842dba572b12d500a6db3, State: Initialized, Role: FOLLOWER
I20260928 02:41:19.169302 23233 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } }
I20260928 02:41:19.169565 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } } }
I20260928 02:41:19.169621 23233 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:19.169680 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:41:19.170360 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.162851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.9.130.61:35945)
0928 02:41:19.162883 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:19.162885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:19.162891 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:41:19.162909 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:19.162910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:19.162925 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:19.162974 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:19.163559 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.164137 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.165006 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.165208 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.165261 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.167722 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.167723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.167723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.167729 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.169194 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.169561 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.169562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.169562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.169620 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.169848 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.169849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.169849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.169951 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.170082 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.170182 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.170259 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 02:41:19.171576 23250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:41:19.171625 23250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:41:19.171866 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:41:19.171913 23165 master_runner.cc:186] Error getting master registration for 127.9.130.62:35811: OK, instance_id { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" instance_seqno: 1790563278836756 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:41:19.174654 23165 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:41:19.180189 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } attempt: 2
I20260928 02:41:19.182080 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.173085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.9.130.61:35945)
0928 02:41:19.173122 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:19.173124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:19.173771 (+ 647us) server_negotiation.cc:407] Connection header received
0928 02:41:19.173837 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:19.173838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:19.173849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:19.173900 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:19.174981 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.175630 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.176566 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.176827 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.176898 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.178419 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.178420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.178421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.178427 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.180139 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.181070 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.181071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.181072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.181126 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.181335 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.181336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.181336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.181462 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.181547 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.181760 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.181834 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:41:19.183017 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.173463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.9.130.60:37877)
0928 02:41:19.173666 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:19.173667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:19.173930 (+ 263us) server_negotiation.cc:407] Connection header received
0928 02:41:19.174022 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:19.174023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:19.174031 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:19.174061 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:19.174713 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.175438 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.176530 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.176816 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.176888 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.178591 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.178593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.178593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.178851 (+ 258us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.181541 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.182037 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.182038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.182038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.182091 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.182282 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.182282 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.182283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.182427 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.182587 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.182854 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.182938 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260928 02:41:19.183472 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.173407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.9.130.62:35811)
0928 02:41:19.173500 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:19.173501 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:19.173508 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:41:19.173525 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:19.173526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:19.173534 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:19.173612 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:19.176600 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.177255 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.178276 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.178582 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.178624 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.180022 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.180024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.180024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.180031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.181606 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.182065 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.182066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.182066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.182120 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.182322 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.182323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.182323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.182457 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.182527 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.183277 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.183349 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 02:41:19 dist-test-slave-kkrh krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563278, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for kudu/127.9.130.60@KRBTEST.COM
I20260928 02:41:19.190284 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.180631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:37877 (local address 127.0.0.1:38214)
0928 02:41:19.180757 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.180767 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:19.180809 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:19.182835 (+ 2026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:19.182836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:19.183015 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:19.183061 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:19.183063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.184474 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.184476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.185244 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:19.185246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.185586 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.185587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.185643 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.185975 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:19.185980 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:19.187192 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:19.189216 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.189216 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.189217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.189380 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.189603 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.189603 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.189604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.189659 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.189973 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:19.189974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:19.190015 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:41:19.190101 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:41:19.190191 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 02:41:19.190312 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.180895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.9.130.60:37877)
0928 02:41:19.182579 (+ 1684us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:19.182581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:19.182591 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:41:19.182701 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:19.182701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:19.182712 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:19.183777 (+ 1065us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:19.183806 (+ 29us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.184399 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.185324 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.185515 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.185563 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.187281 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.187282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.187282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.187289 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.189095 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.189476 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.189477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.189478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.189536 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.189740 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.189741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.189741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.189893 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.189970 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.190125 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.190193 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1663,"thread_start_us":108,"threads_started":1}
I20260928 02:41:19.191721 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963: Bootstrap starting.
I20260928 02:41:19.192358 23083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:19.192629 23083 log.cc:824] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:19.193468 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963: No bootstrap required, opened a new log
I20260928 02:41:19.194818 23083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } }
I20260928 02:41:19.194941 23083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:19.194998 23083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d04c18bf19d4230aa59e04682a40963, State: Initialized, Role: FOLLOWER
I20260928 02:41:19.195147 23083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } }
I20260928 02:41:19.195390 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } } }
I20260928 02:41:19.195452 23083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:19.195472 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:41:19.197167 23272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:41:19.197217 23272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:41:19.198097 23015 master_runner.cc:186] Error getting master registration for 127.9.130.61:35945: OK, instance_id { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" instance_seqno: 1790563278983348 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:41:19.200897 23015 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:41:19.218744 23158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } attempt: 3
Sep 28 02:41:19 dist-test-slave-kkrh krb5kdc[23002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563278, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.61@KRBTEST.COM for kudu/127.9.130.60@KRBTEST.COM
I20260928 02:41:19.227443 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.219056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:37877 (local address 127.0.0.1:38224)
0928 02:41:19.219160 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:19.219172 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:19.219216 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:19.219518 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:19.219519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:19.219758 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:19.219802 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:19.219804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.220830 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.220831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.221673 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:19.221676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.222230 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.222231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:19.222291 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.222824 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:19.222830 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:19.224416 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:19.226376 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.226377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.226378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.226583 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.226807 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:19.226808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:19.226808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:19.226863 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:19.227147 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:19.227148 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:19.227191 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:41:19.227250 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:41:19.227340 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 02:41:19.227715 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.219299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.9.130.60:37877)
0928 02:41:19.219348 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:19.219351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:19.219360 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:41:19.219380 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:19.219381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:19.219390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:19.219462 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:19.219896 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.220689 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.221788 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.222120 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.222230 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.224523 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.224525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.224525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.224532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.226286 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.226660 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.226661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.226661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.226729 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.226937 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.226938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.226938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.227067 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.227123 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.227553 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.227628 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:41:19.229417 23158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18: Bootstrap starting.
I20260928 02:41:19.230154 23158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:19.230459 23158 log.cc:824] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:19.231478 23158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18: No bootstrap required, opened a new log
I20260928 02:41:19.233560 23158 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } }
I20260928 02:41:19.233700 23158 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:19.233735 23158 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0451a68581249e391c85f33e0ed5d18, State: Initialized, Role: FOLLOWER
I20260928 02:41:19.233870 23158 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } }
I20260928 02:41:19.234184 23158 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:19.234150 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } } }
I20260928 02:41:19.234325 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:41:19.235931 23286 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:41:19.236030 23286 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:41:19.237962 23090 master_runner.cc:263] No leader master found from master c0451a68581249e391c85f33e0ed5d18
I20260928 02:41:19.238013 23090 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 02:41:19.334592 23036 tablet.cc:2369] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:41:19.382772 23186 tablet.cc:2369] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:19.390996 23275 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:19.391153 23275 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } }
I20260928 02:41:19.391708 23275 leader_election.cc:288] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d04c18bf19d4230aa59e04682a40963 (127.9.130.62:35811), c7cc90d835a842dba572b12d500a6db3 (127.9.130.60:37877)
I20260928 02:41:19.392059 23057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0451a68581249e391c85f33e0ed5d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d04c18bf19d4230aa59e04682a40963" is_pre_election: true
I20260928 02:41:19.392062 23207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0451a68581249e391c85f33e0ed5d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7cc90d835a842dba572b12d500a6db3" is_pre_election: true
I20260928 02:41:19.392220 23057 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0451a68581249e391c85f33e0ed5d18 in term 0.
I20260928 02:41:19.392227 23207 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0451a68581249e391c85f33e0ed5d18 in term 0.
I20260928 02:41:19.392418 23107 leader_election.cc:302] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d04c18bf19d4230aa59e04682a40963, c0451a68581249e391c85f33e0ed5d18; no voters:
I20260928 02:41:19.392571 23275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:19.392637 23275 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:19.392653 23275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:19.393405 23275 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } }
I20260928 02:41:19.393534 23275 leader_election.cc:288] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [CANDIDATE]: Term 1 election: Requested vote from peers 6d04c18bf19d4230aa59e04682a40963 (127.9.130.62:35811), c7cc90d835a842dba572b12d500a6db3 (127.9.130.60:37877)
I20260928 02:41:19.393662 23057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0451a68581249e391c85f33e0ed5d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d04c18bf19d4230aa59e04682a40963"
I20260928 02:41:19.393689 23207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0451a68581249e391c85f33e0ed5d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7cc90d835a842dba572b12d500a6db3"
I20260928 02:41:19.393743 23057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:19.393778 23207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:19.394431 23057 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0451a68581249e391c85f33e0ed5d18 in term 1.
I20260928 02:41:19.394601 23107 leader_election.cc:302] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d04c18bf19d4230aa59e04682a40963, c0451a68581249e391c85f33e0ed5d18; no voters:
I20260928 02:41:19.394630 23207 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0451a68581249e391c85f33e0ed5d18 in term 1.
I20260928 02:41:19.394701 23275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:19.394868 23275 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [term 1 LEADER]: Becoming Leader. State: Replica: c0451a68581249e391c85f33e0ed5d18, State: Running, Role: LEADER
I20260928 02:41:19.394960 23275 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } }
I20260928 02:41:19.395406 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0451a68581249e391c85f33e0ed5d18. Latest consensus state: current_term: 1 leader_uuid: "c0451a68581249e391c85f33e0ed5d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } } }
I20260928 02:41:19.395501 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:19.395684 23290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:19.395937 23290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:19.397411 23057 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [term 1 FOLLOWER]: Refusing update from remote peer c0451a68581249e391c85f33e0ed5d18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:19.397464 23207 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [term 1 FOLLOWER]: Refusing update from remote peer c0451a68581249e391c85f33e0ed5d18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:19.397713 23275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:19.397837 23288 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:19.398950 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0451a68581249e391c85f33e0ed5d18. Latest consensus state: current_term: 1 leader_uuid: "c0451a68581249e391c85f33e0ed5d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } } }
I20260928 02:41:19.399039 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:41:19.399804 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0451a68581249e391c85f33e0ed5d18. Latest consensus state: current_term: 1 leader_uuid: "c0451a68581249e391c85f33e0ed5d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } } }
I20260928 02:41:19.399917 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:41:19.400328 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0451a68581249e391c85f33e0ed5d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } } }
I20260928 02:41:19.400405 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:19.400524 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0451a68581249e391c85f33e0ed5d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } } }
I20260928 02:41:19.400599 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:41:19.400820 23292 mvcc.cc:205] Tried to move back new op lower bound from 7334147192410820608 to 7334147192403185664. Current Snapshot: MvccSnapshot[applied={T|T < 7334147192410820608}]
I20260928 02:41:19.400970 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0451a68581249e391c85f33e0ed5d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } } }
I20260928 02:41:19.401050 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:19.401077 23290 catalog_manager.cc:1347] Generated new cluster ID: 63b97e401717496bb365b38604771e45
I20260928 02:41:19.401113 23290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:19.401101 23291 mvcc.cc:205] Tried to move back new op lower bound from 7334147192410820608 to 7334147192403185664. Current Snapshot: MvccSnapshot[applied={T|T < 7334147192410820608 or (T in {7334147192410820608})}]
I20260928 02:41:19.408430 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0451a68581249e391c85f33e0ed5d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d04c18bf19d4230aa59e04682a40963" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35811 } } peers { permanent_uuid: "c0451a68581249e391c85f33e0ed5d18" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 35945 } } peers { permanent_uuid: "c7cc90d835a842dba572b12d500a6db3" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 37877 } } }
I20260928 02:41:19.408525 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:41:19.415942 23290 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:19.416599 23290 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:19.425369 23290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0451a68581249e391c85f33e0ed5d18: Generated new TSK 0
I20260928 02:41:19.425567 23290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:41:20.172119 23250 catalog_manager.cc:1259] Loaded cluster ID: 63b97e401717496bb365b38604771e45
I20260928 02:41:20.172202 23250 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3: loading cluster ID for follower catalog manager: success
I20260928 02:41:20.173002 23250 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3: acquiring CA information for follower catalog manager: success
I20260928 02:41:20.173292 23250 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7cc90d835a842dba572b12d500a6db3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:41:20.197640 23272 catalog_manager.cc:1259] Loaded cluster ID: 63b97e401717496bb365b38604771e45
I20260928 02:41:20.197738 23272 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963: loading cluster ID for follower catalog manager: success
I20260928 02:41:20.198520 23272 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963: acquiring CA information for follower catalog manager: success
I20260928 02:41:20.198839 23272 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6d04c18bf19d4230aa59e04682a40963: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:41:20.490972 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 23015
I20260928 02:41:20.498106 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 23090
I20260928 02:41:20.502929 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 23165
2026-09-28T02:41:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2959 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:20 dist-test-slave-kkrh krb5kdc[23305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:20 dist-test-slave-kkrh krb5kdc[23305](info): set up 2 sockets
Sep 28 02:41:20 dist-test-slave-kkrh krb5kdc[23305](info): commencing operation
krb5kdc: starting...
W20260928 02:41:21.535861 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:41:21 dist-test-slave-kkrh krb5kdc[23305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:41:21.647140 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:45321
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:45321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:21.728060 23318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:21.728272 23318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:21.728308 23318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:21.729493 23318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:21.729561 23318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:21.729612 23318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:21.729640 23318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:21.729664 23318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:21.731213 23318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:45321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:45321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:21.731472 23318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:21.731691 23318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:21.736506 23326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:21.736503 23324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:21.736573 23323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:21.736754 23318 server_base.cc:1034] running on GCE node
I20260928 02:41:21.736976 23318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:21.737257 23318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:21.738435 23318 hybrid_clock.cc:648] HybridClock initialized: now 1790563281738419 us; error 24 us; skew 500 ppm
Sep 28 02:41:21 dist-test-slave-kkrh krb5kdc[23305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563281, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:21.740901 23318 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:41:21.741396 23318 webserver.cc:460] Webserver started at http://127.9.130.62:33671/ using document root <none> and password file <none>
I20260928 02:41:21.741611 23318 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:21.741678 23318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:21.741808 23318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:21.742827 23318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "b245a0bd9d794982a458bd6359fcd3d4"
format_stamp: "Formatted at 2026-09-28 02:41:21 on dist-test-slave-kkrh"
server_key: "b1b72f0fe0a67288cfe15b6237de717e"
server_key_iv: "10349e7fd15d4817e87437d32be3ecee"
server_key_version: "encryptionkey@0"
I20260928 02:41:21.743196 23318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "b245a0bd9d794982a458bd6359fcd3d4"
format_stamp: "Formatted at 2026-09-28 02:41:21 on dist-test-slave-kkrh"
server_key: "b1b72f0fe0a67288cfe15b6237de717e"
server_key_iv: "10349e7fd15d4817e87437d32be3ecee"
server_key_version: "encryptionkey@0"
I20260928 02:41:21.744556 23318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:41:21.745277 23333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:21.745441 23318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:41:21.745538 23318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "b245a0bd9d794982a458bd6359fcd3d4"
format_stamp: "Formatted at 2026-09-28 02:41:21 on dist-test-slave-kkrh"
server_key: "b1b72f0fe0a67288cfe15b6237de717e"
server_key_iv: "10349e7fd15d4817e87437d32be3ecee"
server_key_version: "encryptionkey@0"
I20260928 02:41:21.745615 23318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:21.757896 23318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:21.759536 23318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:21.759688 23318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:21.763450 23318 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:45321
I20260928 02:41:21.763499 23385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:45321 every 8 connection(s)
I20260928 02:41:21.763885 23318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:21.764714 23386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:21.766851 23386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4: Bootstrap starting.
I20260928 02:41:21.767406 23386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:21.767688 23386 log.cc:824] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:21.768383 23386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4: No bootstrap required, opened a new log
I20260928 02:41:21.769919 23386 raft_consensus.cc:348] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b245a0bd9d794982a458bd6359fcd3d4" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45321 } }
I20260928 02:41:21.770049 23386 raft_consensus.cc:374] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:21.770071 23386 raft_consensus.cc:729] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b245a0bd9d794982a458bd6359fcd3d4, State: Initialized, Role: FOLLOWER
I20260928 02:41:21.770243 23386 consensus_queue.cc:260] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b245a0bd9d794982a458bd6359fcd3d4" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45321 } }
I20260928 02:41:21.770327 23386 raft_consensus.cc:388] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:21.770375 23386 raft_consensus.cc:482] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:21.770423 23386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:21.770937 23386 raft_consensus.cc:504] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b245a0bd9d794982a458bd6359fcd3d4" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45321 } }
I20260928 02:41:21.771045 23386 leader_election.cc:302] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b245a0bd9d794982a458bd6359fcd3d4; no voters:
I20260928 02:41:21.771238 23386 leader_election.cc:288] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:21.771299 23389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:21.771494 23386 sys_catalog.cc:564] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:21.771499 23389 raft_consensus.cc:686] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [term 1 LEADER]: Becoming Leader. State: Replica: b245a0bd9d794982a458bd6359fcd3d4, State: Running, Role: LEADER
I20260928 02:41:21.771648 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 23318
I20260928 02:41:21.771703 23389 consensus_queue.cc:237] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b245a0bd9d794982a458bd6359fcd3d4" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45321 } }
I20260928 02:41:21.771760 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:41:21.771996 9736 external_mini_cluster.cc:1428] Setting key 9b9d0525ca8c58a2e5cb71481df45b54
I20260928 02:41:21.772096 23391 sys_catalog.cc:455] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b245a0bd9d794982a458bd6359fcd3d4. Latest consensus state: current_term: 1 leader_uuid: "b245a0bd9d794982a458bd6359fcd3d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b245a0bd9d794982a458bd6359fcd3d4" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45321 } } }
I20260928 02:41:21.772220 23391 sys_catalog.cc:458] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:21.772220 23390 sys_catalog.cc:455] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b245a0bd9d794982a458bd6359fcd3d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b245a0bd9d794982a458bd6359fcd3d4" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 45321 } } }
I20260928 02:41:21.772321 23390 sys_catalog.cc:458] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:21.772480 23398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:21.772977 23398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:21.774461 23398 catalog_manager.cc:1347] Generated new cluster ID: 9e5cf4ea38c7470990d4a94b835584e2
I20260928 02:41:21.774556 23398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:41:21 dist-test-slave-kkrh krb5kdc[23305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:21.784085 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:21.772587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.9.130.62:45321)
0928 02:41:21.773376 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:21.773386 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:21.773417 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:41:21.773577 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:21.773585 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:21.773650 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:21.773787 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:21.774542 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:21.775952 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:21.777025 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:21.777635 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:21.777728 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:21.780482 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:21.780488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:21.780488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:21.780502 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:21.782473 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:21.782846 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:21.782849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:21.782849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:21.782969 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:21.783169 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:21.783170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:21.783170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:21.783471 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:21.783645 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:21.783713 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 02:41:21.783881 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":67,"threads_started":1}
I20260928 02:41:21.785291 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:41:21.790771 23398 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:21.791430 23398 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:21.805325 23398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b245a0bd9d794982a458bd6359fcd3d4: Generated new TSK 0
I20260928 02:41:21.805521 23398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:41:21.839133 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 23318
2026-09-28T02:41:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1334 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:21 dist-test-slave-kkrh krb5kdc[23413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:21 dist-test-slave-kkrh krb5kdc[23413](info): set up 2 sockets
Sep 28 02:41:21 dist-test-slave-kkrh krb5kdc[23413](info): commencing operation
krb5kdc: starting...
W20260928 02:41:22.916587 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:41:22 dist-test-slave-kkrh krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:41:23.034644 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:38647
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:38647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:23.114008 23426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:23.114209 23426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:23.114231 23426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:23.115285 23426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:23.115319 23426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:23.115335 23426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:23.115347 23426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:23.115360 23426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:23.116740 23426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:38647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:38647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:23.116923 23426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:23.117079 23426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:23.121784 23431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:23.121815 23432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:23.121811 23434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:23.121969 23426 server_base.cc:1034] running on GCE node
I20260928 02:41:23.122232 23426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:23.122470 23426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:23.123642 23426 hybrid_clock.cc:648] HybridClock initialized: now 1790563283123640 us; error 54 us; skew 500 ppm
Sep 28 02:41:23 dist-test-slave-kkrh krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563283, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:23.126168 23426 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:41:23.126622 23426 webserver.cc:460] Webserver started at http://127.9.130.62:46083/ using document root <none> and password file <none>
I20260928 02:41:23.126874 23426 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:23.126945 23426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:23.127092 23426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:23.128192 23426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "5b2faf845e5249b387d7bc18099eb051"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "05b13f855ddccb90c071e7d7c8e3e229"
server_key_iv: "8d68fee9451ba5270b661b7aab10704a"
server_key_version: "encryptionkey@0"
I20260928 02:41:23.128568 23426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "5b2faf845e5249b387d7bc18099eb051"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "05b13f855ddccb90c071e7d7c8e3e229"
server_key_iv: "8d68fee9451ba5270b661b7aab10704a"
server_key_version: "encryptionkey@0"
I20260928 02:41:23.129822 23426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:23.130479 23441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:23.130650 23426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:41:23.130752 23426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "5b2faf845e5249b387d7bc18099eb051"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "05b13f855ddccb90c071e7d7c8e3e229"
server_key_iv: "8d68fee9451ba5270b661b7aab10704a"
server_key_version: "encryptionkey@0"
I20260928 02:41:23.130823 23426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:23.152904 23426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:23.155230 23426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:23.155398 23426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:23.159341 23426 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:38647
I20260928 02:41:23.159377 23493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:38647 every 8 connection(s)
I20260928 02:41:23.159756 23426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:23.160852 23494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:23.163250 23494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Bootstrap starting.
I20260928 02:41:23.163806 23494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:23.164077 23494 log.cc:824] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:23.164749 23494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: No bootstrap required, opened a new log
I20260928 02:41:23.166493 23494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } }
I20260928 02:41:23.166638 23494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:23.166662 23494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b2faf845e5249b387d7bc18099eb051, State: Initialized, Role: FOLLOWER
I20260928 02:41:23.166806 23494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } }
I20260928 02:41:23.166901 23494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:23.166929 23494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:23.166960 23494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:23.167781 23494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } }
I20260928 02:41:23.167896 23494 leader_election.cc:302] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b2faf845e5249b387d7bc18099eb051; no voters:
I20260928 02:41:23.168025 23494 leader_election.cc:288] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:23.168088 23497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:23.168272 23497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 1 LEADER]: Becoming Leader. State: Replica: 5b2faf845e5249b387d7bc18099eb051, State: Running, Role: LEADER
I20260928 02:41:23.168324 23494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:23.168370 23497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } }
I20260928 02:41:23.168756 23498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b2faf845e5249b387d7bc18099eb051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } } }
I20260928 02:41:23.168788 23499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b2faf845e5249b387d7bc18099eb051. Latest consensus state: current_term: 1 leader_uuid: "5b2faf845e5249b387d7bc18099eb051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } } }
I20260928 02:41:23.168946 23499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:23.169224 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 23426
I20260928 02:41:23.169319 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:41:23.169538 9736 external_mini_cluster.cc:1428] Setting key 2f9b15af77f6e1baea5bcdfde2c9c803
I20260928 02:41:23.169526 23506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:23.168910 23498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:23.170255 23506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:23.171977 23506 catalog_manager.cc:1347] Generated new cluster ID: f080871379f048b6a5e4decfa0cf97d2
I20260928 02:41:23.172050 23506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:41:23 dist-test-slave-kkrh krb5kdc[23413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:23.178748 23506 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:23.179384 23506 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:23.182082 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.170018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.9.130.62:38647)
0928 02:41:23.170208 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:23.170219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:23.170277 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:41:23.170646 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:23.170654 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:23.170711 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:23.170824 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:23.171674 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.173228 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.174273 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.174988 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.175110 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.178085 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:23.178088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:23.178089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:23.178101 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:23.180339 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:23.180720 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:23.180724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:23.180725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:23.180871 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:23.181082 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:23.181083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:23.181084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:23.181411 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:23.181601 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:23.181676 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:41:23.181834 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260928 02:41:23.184867 23506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Generated new TSK 0
I20260928 02:41:23.185048 23506 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:41:23.212694 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38647
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:23.292860 23520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:23.293097 23520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:23.293134 23520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:23.294289 23520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:23.294360 23520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:23.294413 23520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:41:23.295938 23520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:23.296185 23520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:23.296415 23520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:23.302317 23526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:23.302488 23520 server_base.cc:1034] running on GCE node
W20260928 02:41:23.302487 23525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:23.302625 23528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:23.302902 23520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:23.303159 23520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:23.304320 23520 hybrid_clock.cc:648] HybridClock initialized: now 1790563283304299 us; error 37 us; skew 500 ppm
Sep 28 02:41:23 dist-test-slave-kkrh krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563283, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:23.306988 23520 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:41:23.307402 23520 webserver.cc:460] Webserver started at http://127.9.130.1:38461/ using document root <none> and password file <none>
I20260928 02:41:23.307628 23520 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:23.307698 23520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:23.307886 23520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:23.308925 23520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "2822f695eb1f4f06952efe8857ef957e"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "eb87fc2517a056c207a3dd978eb71438"
server_key_iv: "383b9fabf1a47c5b7065a2ea9d34fb41"
server_key_version: "encryptionkey@0"
I20260928 02:41:23.309301 23520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "2822f695eb1f4f06952efe8857ef957e"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "eb87fc2517a056c207a3dd978eb71438"
server_key_iv: "383b9fabf1a47c5b7065a2ea9d34fb41"
server_key_version: "encryptionkey@0"
I20260928 02:41:23.310633 23520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:41:23.311385 23535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:23.311558 23520 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:41:23.311654 23520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "2822f695eb1f4f06952efe8857ef957e"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "eb87fc2517a056c207a3dd978eb71438"
server_key_iv: "383b9fabf1a47c5b7065a2ea9d34fb41"
server_key_version: "encryptionkey@0"
I20260928 02:41:23.311739 23520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:23.321560 23520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:23.324061 23520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:23.324229 23520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:23.324474 23520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:23.324792 23520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:23.324851 23520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:23.324898 23520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:23.324929 23520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:23.331115 23520 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:46747
I20260928 02:41:23.331192 23648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:46747 every 8 connection(s)
I20260928 02:41:23.331540 23520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:23.337945 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 23520
I20260928 02:41:23.338042 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:23.338263 9736 external_mini_cluster.cc:1428] Setting key c1add60f3d8a7ce82d89f7bda49d3e12
Sep 28 02:41:23 dist-test-slave-kkrh krb5kdc[23413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563283, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:23.342317 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.332339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:44685 (local address 127.9.130.62:38647)
0928 02:41:23.332445 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:23.332448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:23.332630 (+ 182us) server_negotiation.cc:407] Connection header received
0928 02:41:23.333189 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:23.333190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:23.333203 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:23.333260 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:23.334345 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.334950 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.336137 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.336382 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.336476 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.339330 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:23.339331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:23.339331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:23.339337 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:23.340838 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:23.341334 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:23.341336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:23.341336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:23.341388 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:23.341670 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:23.341671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:23.341671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:23.341777 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:23.341829 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:23.342151 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 02:41:23.342224 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:41:23.342590 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.332308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:44685)
0928 02:41:23.332497 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:23.332544 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:23.333079 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:23.333347 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:23.333351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:23.334018 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:23.334220 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.334225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.335064 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.335065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.336034 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.336036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.336487 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.336488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.336620 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.337202 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:23.337210 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:23.339244 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:23.340945 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:23.340952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:23.340954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:23.341255 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:23.341460 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:23.341461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:23.341462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:23.341605 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:23.341842 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:23.341848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:23.342017 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 02:41:23.342130 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 02:41:23.342309 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260928 02:41:23.343109 23649 heartbeater.cc:347] Connected to a master server at 127.9.130.62:38647
I20260928 02:41:23.343225 23649 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:23.343420 23649 heartbeater.cc:511] Master 127.9.130.62:38647 requested a full tablet report, sending...
I20260928 02:41:23.344094 23458 ts_manager.cc:194] Registered new tserver with Master: 2822f695eb1f4f06952efe8857ef957e (127.9.130.1:46747)
I20260928 02:41:23.344859 23458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:44685
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:41:23.364147 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38647
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:23.444425 23654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:23.444658 23654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:23.444694 23654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:23.445845 23654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:23.445916 23654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:23.445960 23654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:41:23.447407 23654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:23.447674 23654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:23.447956 23654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:23.452975 23660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:23.452965 23662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:23.452965 23659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:23.453176 23654 server_base.cc:1034] running on GCE node
I20260928 02:41:23.453436 23654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:23.453662 23654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:23.454805 23654 hybrid_clock.cc:648] HybridClock initialized: now 1790563283454797 us; error 33 us; skew 500 ppm
Sep 28 02:41:23 dist-test-slave-kkrh krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563283, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:23.457410 23654 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:41:23.457873 23654 webserver.cc:460] Webserver started at http://127.9.130.2:39667/ using document root <none> and password file <none>
I20260928 02:41:23.458119 23654 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:23.458185 23654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:23.458338 23654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:23.459369 23654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "68b2054f0b8949de8d7a09b1ba8309b2"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "7984499ae22a0a0b034baa57a8de60e5"
server_key_iv: "8d0bc787deb44d4ef91978ef538980a6"
server_key_version: "encryptionkey@0"
I20260928 02:41:23.459720 23654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "68b2054f0b8949de8d7a09b1ba8309b2"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "7984499ae22a0a0b034baa57a8de60e5"
server_key_iv: "8d0bc787deb44d4ef91978ef538980a6"
server_key_version: "encryptionkey@0"
I20260928 02:41:23.461043 23654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:23.461844 23669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:23.462042 23654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:41:23.462136 23654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "68b2054f0b8949de8d7a09b1ba8309b2"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "7984499ae22a0a0b034baa57a8de60e5"
server_key_iv: "8d0bc787deb44d4ef91978ef538980a6"
server_key_version: "encryptionkey@0"
I20260928 02:41:23.462221 23654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:23.474460 23654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:23.477008 23654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:23.477180 23654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:23.477417 23654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:23.477742 23654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:23.477797 23654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:23.477843 23654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:23.477873 23654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:23.483884 23654 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:34443
I20260928 02:41:23.483950 23782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:34443 every 8 connection(s)
I20260928 02:41:23.484297 23654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:23.488502 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 23654
I20260928 02:41:23.488629 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:23.488797 9736 external_mini_cluster.cc:1428] Setting key 53ae63b0c80020212961807d82f44acf
Sep 28 02:41:23 dist-test-slave-kkrh krb5kdc[23413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563283, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:23.494253 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.485000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:46115 (local address 127.9.130.62:38647)
0928 02:41:23.485048 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:23.485052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:23.485270 (+ 218us) server_negotiation.cc:407] Connection header received
0928 02:41:23.485865 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:23.485867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:23.485878 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:23.485938 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:23.486956 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.487439 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.488556 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.488893 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.488964 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.491328 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:23.491330 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:23.491330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:23.491338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:23.492819 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:23.493303 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:23.493304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:23.493305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:23.493374 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:23.493627 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:23.493627 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:23.493628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:23.493742 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:23.493816 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:23.494082 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 02:41:23.494162 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:41:23.494448 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.484958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:46115)
0928 02:41:23.485119 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:23.485164 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:23.485710 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:23.486005 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:23.486009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:23.486669 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:23.486873 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.486878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.487517 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.487518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.488467 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.488470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.488991 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.488992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.489153 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.489795 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:23.489804 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:23.491231 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:23.492917 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:23.492923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:23.492925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:23.493216 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:23.493441 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:23.493441 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:23.493442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:23.493554 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:23.493806 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:23.493813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:23.493954 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 02:41:23.494063 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:41:23.494218 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260928 02:41:23.495078 23783 heartbeater.cc:347] Connected to a master server at 127.9.130.62:38647
I20260928 02:41:23.495177 23783 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:23.495348 23783 heartbeater.cc:511] Master 127.9.130.62:38647 requested a full tablet report, sending...
I20260928 02:41:23.495767 23458 ts_manager.cc:194] Registered new tserver with Master: 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443)
I20260928 02:41:23.496376 23458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:46115
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:41:23.515383 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38647
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:23.594693 23788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:23.594887 23788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:23.594910 23788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:23.596104 23788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:23.596143 23788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:23.596169 23788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:41:23.597592 23788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:23.597838 23788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:23.598059 23788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:23.603014 23793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:23.603049 23788 server_base.cc:1034] running on GCE node
W20260928 02:41:23.603008 23796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:23.603008 23794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:23.603459 23788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:23.603647 23788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:23.604789 23788 hybrid_clock.cc:648] HybridClock initialized: now 1790563283604765 us; error 42 us; skew 500 ppm
Sep 28 02:41:23 dist-test-slave-kkrh krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563283, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:23.607270 23788 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:41:23.607779 23788 webserver.cc:460] Webserver started at http://127.9.130.3:43839/ using document root <none> and password file <none>
I20260928 02:41:23.608047 23788 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:23.608114 23788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:23.608266 23788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:23.609300 23788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "9466be8e5d8a442c954713facc68b7e6"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "c64e09957cb066becf774a3f6bcbae61"
server_key_iv: "09fd882fe6590d734eb941995af858e4"
server_key_version: "encryptionkey@0"
I20260928 02:41:23.609656 23788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "9466be8e5d8a442c954713facc68b7e6"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "c64e09957cb066becf774a3f6bcbae61"
server_key_iv: "09fd882fe6590d734eb941995af858e4"
server_key_version: "encryptionkey@0"
I20260928 02:41:23.610967 23788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:41:23.611727 23803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:23.611925 23788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:23.612021 23788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "9466be8e5d8a442c954713facc68b7e6"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "c64e09957cb066becf774a3f6bcbae61"
server_key_iv: "09fd882fe6590d734eb941995af858e4"
server_key_version: "encryptionkey@0"
I20260928 02:41:23.612102 23788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:23.636600 23788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:23.638952 23788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:23.639113 23788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:23.639374 23788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:23.639679 23788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:23.639740 23788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:23.639806 23788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:23.639859 23788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:23.645584 23788 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:34727
I20260928 02:41:23.645635 23916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:34727 every 8 connection(s)
I20260928 02:41:23.645941 23788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:23.650134 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 23788
I20260928 02:41:23.650249 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:23.650451 9736 external_mini_cluster.cc:1428] Setting key ec6423bf569a4c94e55d601541e1844b
Sep 28 02:41:23 dist-test-slave-kkrh krb5kdc[23413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563283, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:23.655148 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.646660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:45643 (local address 127.9.130.62:38647)
0928 02:41:23.646723 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:23.646725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:23.646902 (+ 177us) server_negotiation.cc:407] Connection header received
0928 02:41:23.647440 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:23.647442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:23.647452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:23.647501 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:23.648489 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.648983 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.649952 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.650135 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.650189 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.652647 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:23.652648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:23.652648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:23.652655 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:23.654053 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:23.654401 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:23.654402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:23.654402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:23.654455 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:23.654657 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:23.654657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:23.654657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:23.654755 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:23.654803 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:23.655025 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 02:41:23.655081 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:41:23.655390 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.646563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:45643)
0928 02:41:23.646793 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:23.646829 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:23.647345 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:23.647566 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:23.647569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:23.648223 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:23.648410 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.648415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.649059 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.649060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.649869 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.649871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.650221 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.650222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.650375 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.650963 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:23.650973 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:23.652566 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:23.654121 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:23.654126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:23.654128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:23.654331 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:23.654513 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:23.654513 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:23.654513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:23.654604 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:23.654807 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:23.654811 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:23.654917 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 02:41:23.654997 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:41:23.655113 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":105,"threads_started":1}
I20260928 02:41:23.655870 23917 heartbeater.cc:347] Connected to a master server at 127.9.130.62:38647
I20260928 02:41:23.655972 23917 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:23.656136 23917 heartbeater.cc:511] Master 127.9.130.62:38647 requested a full tablet report, sending...
I20260928 02:41:23.656575 23458 ts_manager.cc:194] Registered new tserver with Master: 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727)
I20260928 02:41:23.656950 23458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:45643
I20260928 02:41:23.662122 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:41:23.669209 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.663490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.9.130.62:38647)
0928 02:41:23.663548 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:23.663550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:23.663647 (+ 97us) server_negotiation.cc:407] Connection header received
0928 02:41:23.663688 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:23.663689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:23.663699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:23.663750 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:23.664316 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.664766 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.665525 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.665701 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.665751 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.666875 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:23.666876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:23.666876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:23.666882 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:23.668131 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:23.668438 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:23.668439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:23.668439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:23.668488 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:23.668634 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:23.668635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:23.668635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:23.668858 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:23.668933 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:23.669025 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 02:41:23.669079 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:41:23.670673 23458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:23.672145 23458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:23.678084 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.674107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46747 (local address 127.0.0.1:37028)
0928 02:41:23.674446 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:23.674454 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:23.674487 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:23.674657 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:23.674660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:23.674662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:23.674734 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.674735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.676094 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.676095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.677086 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.677089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.677801 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.677802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.677877 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.677890 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:41:23.677965 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:41:23.678012 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":55,"threads_started":1}
I20260928 02:41:23.678108 23927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.674210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.9.130.1:46747)
0928 02:41:23.674478 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:23.674483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:23.674495 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:41:23.674539 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:23.674541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:23.674558 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:23.674646 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:23.674795 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.675993 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.677158 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.677715 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.677788 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.677841 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:23.677987 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 02:41:23.678029 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":77,"threads_started":1}
I20260928 02:41:23.678606 23925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.674086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:34727 (local address 127.0.0.1:35752)
0928 02:41:23.674214 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:23.674253 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:23.674315 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:23.674778 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:23.674778 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:23.674780 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:23.674825 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.674827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.676143 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.676144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.677463 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.677466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.678362 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.678363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.678433 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.678436 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:41:23.678490 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:41:23.678529 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":74,"thread_start_us":51,"threads_started":1}
I20260928 02:41:23.678629 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.674274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.9.130.3:34727)
0928 02:41:23.674611 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:23.674617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:23.674626 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:41:23.674665 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:23.674667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:23.674681 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:23.674780 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:23.674968 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.676029 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.677553 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.678282 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.678360 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.678412 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:23.678519 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 02:41:23.678557 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":72,"threads_started":1}
I20260928 02:41:23.678856 23583 tablet_service.cc:1467] Processing CreateTablet for tablet 7a90b6711b874ee0a7c878971d9b1de0 (DEFAULT_TABLE table=test-table [id=a060a1a3efb7476abc2dca3977085550]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:23.679184 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.674276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:34443 (local address 127.0.0.1:42824)
0928 02:41:23.674867 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:23.674878 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:23.674926 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:23.675255 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:23.675256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:23.675257 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:23.675294 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.675295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.676217 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.676218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.677465 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.677466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.678920 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.678921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.678980 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.678984 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:41:23.679079 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:41:23.679117 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":101,"thread_start_us":45,"threads_started":1}
I20260928 02:41:23.679220 23583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a90b6711b874ee0a7c878971d9b1de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:23.679301 23851 tablet_service.cc:1467] Processing CreateTablet for tablet 7a90b6711b874ee0a7c878971d9b1de0 (DEFAULT_TABLE table=test-table [id=a060a1a3efb7476abc2dca3977085550]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:23.679605 23851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a90b6711b874ee0a7c878971d9b1de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:23.679620 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.674471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.9.130.2:34443)
0928 02:41:23.674635 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:23.674639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:23.674923 (+ 284us) server_negotiation.cc:407] Connection header received
0928 02:41:23.675080 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:23.675083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:23.675098 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:23.675168 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:23.675365 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.676120 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.677564 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.678522 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.678578 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.678628 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:23.679493 (+ 865us) server_negotiation.cc:299] Negotiation successful
0928 02:41:23.679535 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260928 02:41:23.680400 23717 tablet_service.cc:1467] Processing CreateTablet for tablet 7a90b6711b874ee0a7c878971d9b1de0 (DEFAULT_TABLE table=test-table [id=a060a1a3efb7476abc2dca3977085550]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:23.680696 23717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a90b6711b874ee0a7c878971d9b1de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:23.682065 23931 tablet_bootstrap.cc:492] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Bootstrap starting.
I20260928 02:41:23.682768 23932 tablet_bootstrap.cc:492] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6: Bootstrap starting.
I20260928 02:41:23.682948 23931 tablet_bootstrap.cc:654] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:23.683445 23932 tablet_bootstrap.cc:654] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:23.683521 23931 log.cc:824] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:23.683625 23933 tablet_bootstrap.cc:492] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: Bootstrap starting.
I20260928 02:41:23.684021 23932 log.cc:824] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:23.684274 23933 tablet_bootstrap.cc:654] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:23.684530 23931 tablet_bootstrap.cc:492] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: No bootstrap required, opened a new log
I20260928 02:41:23.684612 23931 ts_tablet_manager.cc:1397] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:41:23.684758 23932 tablet_bootstrap.cc:492] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6: No bootstrap required, opened a new log
I20260928 02:41:23.684834 23932 ts_tablet_manager.cc:1397] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:23.684834 23933 log.cc:824] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:23.685508 23933 tablet_bootstrap.cc:492] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: No bootstrap required, opened a new log
I20260928 02:41:23.685578 23933 ts_tablet_manager.cc:1397] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:41:23.686374 23931 raft_consensus.cc:348] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } }
I20260928 02:41:23.686546 23931 raft_consensus.cc:374] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:23.686587 23931 raft_consensus.cc:729] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2822f695eb1f4f06952efe8857ef957e, State: Initialized, Role: FOLLOWER
I20260928 02:41:23.686697 23931 consensus_queue.cc:260] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } }
I20260928 02:41:23.686817 23932 raft_consensus.cc:348] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } }
I20260928 02:41:23.686987 23932 raft_consensus.cc:374] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:23.686977 23931 ts_tablet_manager.cc:1428] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:23.687031 23932 raft_consensus.cc:729] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9466be8e5d8a442c954713facc68b7e6, State: Initialized, Role: FOLLOWER
I20260928 02:41:23.687114 23649 heartbeater.cc:503] Master 127.9.130.62:38647 was elected leader, sending a full tablet report...
I20260928 02:41:23.687151 23932 consensus_queue.cc:260] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } }
I20260928 02:41:23.687439 23917 heartbeater.cc:503] Master 127.9.130.62:38647 was elected leader, sending a full tablet report...
I20260928 02:41:23.687497 23933 raft_consensus.cc:348] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } }
I20260928 02:41:23.687623 23932 ts_tablet_manager.cc:1428] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:23.687644 23933 raft_consensus.cc:374] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:23.687675 23933 raft_consensus.cc:729] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b2054f0b8949de8d7a09b1ba8309b2, State: Initialized, Role: FOLLOWER
I20260928 02:41:23.687779 23933 consensus_queue.cc:260] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } }
I20260928 02:41:23.688043 23933 ts_tablet_manager.cc:1428] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 02:41:23.688153 23783 heartbeater.cc:503] Master 127.9.130.62:38647 was elected leader, sending a full tablet report...
W20260928 02:41:23.734858 23784 tablet.cc:2369] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:41:23.832296 23650 tablet.cc:2369] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:23.888075 23938 raft_consensus.cc:482] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:23.888222 23938 raft_consensus.cc:504] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } }
I20260928 02:41:23.888754 23938 leader_election.cc:288] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2822f695eb1f4f06952efe8857ef957e (127.9.130.1:46747), 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443)
I20260928 02:41:23.891847 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.888980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46747 (local address 127.9.130.3:49689)
0928 02:41:23.889049 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:23.889063 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:23.889155 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:23.889354 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:23.889355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:23.889357 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:23.889447 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.889450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.890194 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.890195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.891018 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.891020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.891543 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.891544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.891634 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.891638 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:41:23.891691 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 02:41:23.891748 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":50}
I20260928 02:41:23.891889 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.889240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:56841 (local address 127.9.130.2:34443)
0928 02:41:23.889286 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:23.889288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:23.889417 (+ 129us) server_negotiation.cc:407] Connection header received
0928 02:41:23.889493 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:23.889494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:23.889508 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:23.889560 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:23.889715 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.890200 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.891125 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.891515 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.891584 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.891594 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:23.891759 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:41:23.891798 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:41:23.891855 23927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.889109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:49689 (local address 127.9.130.1:46747)
0928 02:41:23.889179 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:23.889182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:23.889205 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:41:23.889260 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:23.889261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:23.889274 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:23.889329 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:23.889558 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.890107 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.891114 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.891462 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.891513 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.891521 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:23.891713 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:41:23.891754 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:41:23.891945 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.889151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:34443 (local address 127.9.130.3:56841)
0928 02:41:23.889322 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:23.889333 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:23.889399 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:23.889574 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:23.889575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:23.889577 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:23.889643 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.889645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.890280 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.890281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.891021 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:23.891023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.891603 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.891604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:23.891655 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.891658 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:41:23.891732 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:41:23.891765 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260928 02:41:23.892340 23603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a90b6711b874ee0a7c878971d9b1de0" candidate_uuid: "9466be8e5d8a442c954713facc68b7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2822f695eb1f4f06952efe8857ef957e" is_pre_election: true
I20260928 02:41:23.892340 23737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a90b6711b874ee0a7c878971d9b1de0" candidate_uuid: "9466be8e5d8a442c954713facc68b7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" is_pre_election: true
I20260928 02:41:23.892488 23737 raft_consensus.cc:2454] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9466be8e5d8a442c954713facc68b7e6 in term 0.
I20260928 02:41:23.892488 23603 raft_consensus.cc:2454] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9466be8e5d8a442c954713facc68b7e6 in term 0.
I20260928 02:41:23.892702 23805 leader_election.cc:302] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2822f695eb1f4f06952efe8857ef957e, 9466be8e5d8a442c954713facc68b7e6; no voters:
I20260928 02:41:23.892872 23938 raft_consensus.cc:2781] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:23.892949 23938 raft_consensus.cc:482] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:23.892966 23938 raft_consensus.cc:3037] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:23.893707 23938 raft_consensus.cc:504] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } }
I20260928 02:41:23.893837 23938 leader_election.cc:288] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [CANDIDATE]: Term 1 election: Requested vote from peers 2822f695eb1f4f06952efe8857ef957e (127.9.130.1:46747), 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443)
I20260928 02:41:23.894032 23603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a90b6711b874ee0a7c878971d9b1de0" candidate_uuid: "9466be8e5d8a442c954713facc68b7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2822f695eb1f4f06952efe8857ef957e"
I20260928 02:41:23.894048 23737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a90b6711b874ee0a7c878971d9b1de0" candidate_uuid: "9466be8e5d8a442c954713facc68b7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68b2054f0b8949de8d7a09b1ba8309b2"
I20260928 02:41:23.894110 23603 raft_consensus.cc:3037] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:23.894119 23737 raft_consensus.cc:3037] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:23.894977 23603 raft_consensus.cc:2454] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9466be8e5d8a442c954713facc68b7e6 in term 1.
I20260928 02:41:23.894977 23737 raft_consensus.cc:2454] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9466be8e5d8a442c954713facc68b7e6 in term 1.
I20260928 02:41:23.895150 23805 leader_election.cc:302] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2822f695eb1f4f06952efe8857ef957e, 9466be8e5d8a442c954713facc68b7e6; no voters:
I20260928 02:41:23.895247 23938 raft_consensus.cc:2781] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:23.895396 23938 raft_consensus.cc:686] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [term 1 LEADER]: Becoming Leader. State: Replica: 9466be8e5d8a442c954713facc68b7e6, State: Running, Role: LEADER
I20260928 02:41:23.895514 23938 consensus_queue.cc:237] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } }
W20260928 02:41:23.896440 23918 tablet.cc:2369] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:23.896449 23458 catalog_manager.cc:5495] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3). New cstate: current_term: 1 leader_uuid: "9466be8e5d8a442c954713facc68b7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } health_report { overall_health: UNKNOWN } } }
I20260928 02:41:23.903189 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.900874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.9.130.3:34727)
0928 02:41:23.900933 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:23.900936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:23.900944 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:41:23.900974 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:23.900975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:23.900992 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:23.901060 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:23.901210 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.901654 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.902402 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:23.902611 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:23.902687 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:23.902852 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:23.902956 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:23.903051 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:23.903070 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 02:41:23.903088 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:41:23.904852 23603 raft_consensus.cc:1261] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Refusing update from remote peer 9466be8e5d8a442c954713facc68b7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:23.904917 23737 raft_consensus.cc:1261] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 1 FOLLOWER]: Refusing update from remote peer 9466be8e5d8a442c954713facc68b7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:23.905272 23941 consensus_queue.cc:1035] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:23.905381 23938 consensus_queue.cc:1035] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:23.907790 23946 mvcc.cc:205] Tried to move back new op lower bound from 7334147210872475648 to 7334147210837630976. Current Snapshot: MvccSnapshot[applied={T|T < 7334147210872475648}]
I20260928 02:41:23.907965 23945 mvcc.cc:205] Tried to move back new op lower bound from 7334147210872475648 to 7334147210837630976. Current Snapshot: MvccSnapshot[applied={T|T < 7334147210872475648}]
I20260928 02:41:23.909864 23458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:23.909966 23458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:23.910633 23458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Sending DeleteTablet for 3 replicas of tablet 7a90b6711b874ee0a7c878971d9b1de0
I20260928 02:41:23.910939 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 23520
I20260928 02:41:23.910974 23851 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a90b6711b874ee0a7c878971d9b1de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:23 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:35752
I20260928 02:41:23.910977 23583 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a90b6711b874ee0a7c878971d9b1de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:23 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:37028
I20260928 02:41:23.911160 23717 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a90b6711b874ee0a7c878971d9b1de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:23 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:42824
I20260928 02:41:23.911657 23954 tablet_replica.cc:330] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6: stopping tablet replica
I20260928 02:41:23.911723 23954 raft_consensus.cc:2229] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:23.911798 23954 raft_consensus.cc:2258] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:23.912181 23954 ts_tablet_manager.cc:1905] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:23.913450 23954 ts_tablet_manager.cc:1918] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:23.913523 23954 log.cc:1196] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/7a90b6711b874ee0a7c878971d9b1de0
I20260928 02:41:23.913787 23954 ts_tablet_manager.cc:1939] T 7a90b6711b874ee0a7c878971d9b1de0 P 9466be8e5d8a442c954713facc68b7e6: Deleting consensus metadata
I20260928 02:41:23.914172 23442 catalog_manager.cc:4841] TS 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727): tablet 7a90b6711b874ee0a7c878971d9b1de0 (table test-table [id=a060a1a3efb7476abc2dca3977085550]) successfully deleted
I20260928 02:41:23.915932 23955 tablet_replica.cc:330] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: stopping tablet replica
I20260928 02:41:23.916039 23955 raft_consensus.cc:2229] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:23.916139 23955 raft_consensus.cc:2258] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:23.916501 23955 ts_tablet_manager.cc:1905] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:41:23.917461 23443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:46747 (error 108)
I20260928 02:41:23.917706 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 23654
I20260928 02:41:23.917920 23955 ts_tablet_manager.cc:1918] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:23.917975 23955 log.cc:1196] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/7a90b6711b874ee0a7c878971d9b1de0
I20260928 02:41:23.920179 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.920053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46747 (local address 127.0.0.1:37032)
0928 02:41:23.920106 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:23.920142 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:46747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:41:23.920342 23443 catalog_manager.cc:4584] TS 2822f695eb1f4f06952efe8857ef957e (127.9.130.1:46747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a90b6711b874ee0a7c878971d9b1de0: Network error: Client connection negotiation failed: client connection to 127.9.130.1:46747: connect: Connection refused (error 111)
I20260928 02:41:23.923833 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 23788
I20260928 02:41:23.924789 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:23.924630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:34443 (local address 127.0.0.1:42832)
0928 02:41:23.924741 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:23.924765 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.2:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 02:41:23.928236 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 23426
I20260928 02:41:23.933336 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:38647
--webserver_interface=127.9.130.62
--webserver_port=46083
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:38647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:24.012121 23957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:24.012310 23957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:24.012331 23957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:24.013471 23957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:24.013522 23957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:24.013537 23957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:24.013550 23957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:24.013600 23957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:24.015013 23957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:38647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:38647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=46083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:24.015246 23957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:24.015455 23957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:24.024101 23964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.024101 23966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.024133 23957 server_base.cc:1034] running on GCE node
W20260928 02:41:24.024101 23963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.024583 23957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:24.024837 23957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:24.025976 23957 hybrid_clock.cc:648] HybridClock initialized: now 1790563284025963 us; error 34 us; skew 500 ppm
Sep 28 02:41:24 dist-test-slave-kkrh krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:24.031060 23957 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:41:24.031494 23957 webserver.cc:460] Webserver started at http://127.9.130.62:46083/ using document root <none> and password file <none>
I20260928 02:41:24.031729 23957 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:24.031806 23957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:24.033164 23957 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:24.033977 23973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.034188 23957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:24.034291 23957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "5b2faf845e5249b387d7bc18099eb051"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "05b13f855ddccb90c071e7d7c8e3e229"
server_key_iv: "8d68fee9451ba5270b661b7aab10704a"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.034670 23957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:24.041399 23957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:24.043576 23957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:24.043736 23957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:24.047286 23957 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:38647
I20260928 02:41:24.047338 24025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:38647 every 8 connection(s)
I20260928 02:41:24.047717 23957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:24.049366 24026 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:41:24.050016 24026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Bootstrap starting.
I20260928 02:41:24.057823 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 23957
I20260928 02:41:24.058130 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:46747
--local_ip_for_outbound_sockets=127.9.130.1
--tserver_master_addrs=127.9.130.62:38647
--webserver_port=38461
--webserver_interface=127.9.130.1
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:41:24.058306 24026 log.cc:824] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:24.060930 24026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:41:24.061230 24026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Bootstrap complete.
I20260928 02:41:24.063736 24026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } }
I20260928 02:41:24.064042 24026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b2faf845e5249b387d7bc18099eb051, State: Initialized, Role: FOLLOWER
I20260928 02:41:24.064234 24026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } }
I20260928 02:41:24.064324 24026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:24.064404 24026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:24.064455 24026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:41:24.065435 24026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } }
I20260928 02:41:24.065553 24026 leader_election.cc:302] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b2faf845e5249b387d7bc18099eb051; no voters:
I20260928 02:41:24.065735 24026 leader_election.cc:288] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:41:24.065773 24030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:41:24.065891 24030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 2 LEADER]: Becoming Leader. State: Replica: 5b2faf845e5249b387d7bc18099eb051, State: Running, Role: LEADER
I20260928 02:41:24.065992 24026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:24.066073 24030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } }
I20260928 02:41:24.066321 24030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b2faf845e5249b387d7bc18099eb051. Latest consensus state: current_term: 2 leader_uuid: "5b2faf845e5249b387d7bc18099eb051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } } }
I20260928 02:41:24.066406 24030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:24.066327 24032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5b2faf845e5249b387d7bc18099eb051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } } }
I20260928 02:41:24.066548 24032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:24.067710 24045 catalog_manager.cc:1259] Loaded cluster ID: f080871379f048b6a5e4decfa0cf97d2
I20260928 02:41:24.067761 24045 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: loading cluster ID for follower catalog manager: success
W20260928 02:41:24.067993 24045 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260928 02:41:24.068040 24045 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260928 02:41:24.068073 24036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:24.068410 24036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:24.068485 24036 catalog_manager.cc:1259] Loaded cluster ID: f080871379f048b6a5e4decfa0cf97d2
I20260928 02:41:24.068514 24036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:24.068678 24036 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 02:41:24.073333 24036 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 02:41:24.075778 24036 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:24.076014 24036 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Loaded TSK: 0
I20260928 02:41:24.076241 24036 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:41:24.139859 24029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:24.140087 24029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:24.140126 24029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:24.141245 24029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:24.141309 24029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:24.141355 24029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:41:24.143077 24029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:46747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=38461
--tserver_master_addrs=127.9.130.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:24.143340 24029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:24.143565 24029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:24.149423 24056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.149430 24054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.149487 24053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.149679 24029 server_base.cc:1034] running on GCE node
I20260928 02:41:24.149893 24029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:24.150158 24029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:24.151345 24029 hybrid_clock.cc:648] HybridClock initialized: now 1790563284151317 us; error 42 us; skew 500 ppm
Sep 28 02:41:24 dist-test-slave-kkrh krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:24.154418 24029 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:41:24.154963 24029 webserver.cc:460] Webserver started at http://127.9.130.1:38461/ using document root <none> and password file <none>
I20260928 02:41:24.155203 24029 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:24.155282 24029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:24.156984 24029 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:24.157728 24063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.157917 24029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:24.158026 24029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "2822f695eb1f4f06952efe8857ef957e"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "eb87fc2517a056c207a3dd978eb71438"
server_key_iv: "383b9fabf1a47c5b7065a2ea9d34fb41"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.158363 24029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:24.174372 24029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:24.176739 24029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:24.176883 24029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:24.177135 24029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:24.177580 24070 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:41:24.179438 24029 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:41:24.179507 24029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:41:24.179548 24029 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:41:24.180223 24029 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:41:24.180279 24029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:41:24.180347 24070 tablet_bootstrap.cc:492] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Bootstrap starting.
I20260928 02:41:24.187767 24029 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:46747
I20260928 02:41:24.187832 24177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:46747 every 8 connection(s)
I20260928 02:41:24.188309 24029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:24.191365 24070 log.cc:824] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:24.193610 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 24029
I20260928 02:41:24.193696 24070 tablet_bootstrap.cc:492] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:41:24.193902 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:34443
--local_ip_for_outbound_sockets=127.9.130.2
--tserver_master_addrs=127.9.130.62:38647
--webserver_port=39667
--webserver_interface=127.9.130.2
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:41:24.193984 24070 tablet_bootstrap.cc:492] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Bootstrap complete.
I20260928 02:41:24.195132 24070 ts_tablet_manager.cc:1397] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260928 02:41:24.197214 24070 raft_consensus.cc:348] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } }
I20260928 02:41:24.197582 24070 raft_consensus.cc:729] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2822f695eb1f4f06952efe8857ef957e, State: Initialized, Role: FOLLOWER
I20260928 02:41:24.197719 24070 consensus_queue.cc:260] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } }
I20260928 02:41:24.198002 24070 ts_tablet_manager.cc:1428] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
Sep 28 02:41:24 dist-test-slave-kkrh krb5kdc[23413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:24.203154 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.189083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:38161)
0928 02:41:24.189258 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.189309 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.190114 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.190631 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.190637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.191027 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:24.191326 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.191332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.193024 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.193025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.194348 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.194351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.196178 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.196180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.196312 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.196978 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:24.196995 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:24.198654 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:24.201161 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.201168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.201171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.201451 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.201770 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.201771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.201772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.201957 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.202476 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:24.202483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:24.202637 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.202755 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.202934 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260928 02:41:24.203234 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.189350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:38161 (local address 127.9.130.62:38647)
0928 02:41:24.189543 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.189551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.189586 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:41:24.190350 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.190357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.190413 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.190554 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:24.191452 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.192895 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.195304 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.195949 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.196064 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.198759 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:24.198763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:24.198763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:24.198776 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:24.201064 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.201548 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.201552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.201552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.201684 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.202035 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.202036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.202037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.202374 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:24.202472 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.202838 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.202994 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
I20260928 02:41:24.204052 24178 heartbeater.cc:347] Connected to a master server at 127.9.130.62:38647
I20260928 02:41:24.204161 24178 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:24.204339 24178 heartbeater.cc:511] Master 127.9.130.62:38647 requested a full tablet report, sending...
I20260928 02:41:24.205150 23990 ts_manager.cc:194] Registered new tserver with Master: 2822f695eb1f4f06952efe8857ef957e (127.9.130.1:46747)
I20260928 02:41:24.206475 23990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:38161
I20260928 02:41:24.207482 24178 heartbeater.cc:503] Master 127.9.130.62:38647 was elected leader, sending a full tablet report...
Sep 28 02:41:24 dist-test-slave-kkrh krb5kdc[23413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for kudu/127.9.130.1@KRBTEST.COM
I20260928 02:41:24.214080 24186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.205916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.9.130.1:46747)
0928 02:41:24.206272 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.206276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.206283 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:41:24.206308 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.206309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.206320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.206392 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:24.206896 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.207868 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.208674 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.209000 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.209090 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.211250 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:24.211258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:24.211258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:24.211265 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:24.212805 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.213222 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.213225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.213225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.213286 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.213484 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.213485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.213485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.213627 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:24.213715 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.213895 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.213986 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":75,"threads_started":1}
I20260928 02:41:24.214080 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.205838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46747 (local address 127.0.0.1:37038)
0928 02:41:24.206090 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.206120 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.206175 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.206408 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.206413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.206766 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:24.206831 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.206833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.207990 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.207991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.208583 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.208585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.209088 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.209089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.209168 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.209672 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:24.209676 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:24.211156 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:24.212907 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.212913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.212915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.213126 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.213355 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.213356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.213357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.213420 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.213707 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:24.213713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:24.213764 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.213860 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.213986 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":121,"threads_started":1}
I20260928 02:41:24.214576 24112 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a90b6711b874ee0a7c878971d9b1de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:23 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:37038
I20260928 02:41:24.215118 24187 tablet_replica.cc:330] stopping tablet replica
I20260928 02:41:24.215243 24187 raft_consensus.cc:2229] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:24.215335 24187 raft_consensus.cc:2258] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:24.215626 24187 ts_tablet_manager.cc:1905] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:24.217658 24187 ts_tablet_manager.cc:1918] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:24.217751 24187 log.cc:1196] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal/wals/7a90b6711b874ee0a7c878971d9b1de0
I20260928 02:41:24.217963 24187 ts_tablet_manager.cc:1939] T 7a90b6711b874ee0a7c878971d9b1de0 P 2822f695eb1f4f06952efe8857ef957e: Deleting consensus metadata
I20260928 02:41:24.218451 23975 catalog_manager.cc:4841] TS 2822f695eb1f4f06952efe8857ef957e (127.9.130.1:46747): tablet 7a90b6711b874ee0a7c878971d9b1de0 (table test-table [id=a060a1a3efb7476abc2dca3977085550]) successfully deleted
W20260928 02:41:24.281522 24183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:24.281744 24183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:24.281780 24183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:24.282917 24183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:24.282980 24183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:24.283025 24183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:41:24.284524 24183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:34443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=39667
--tserver_master_addrs=127.9.130.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:24.284794 24183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:24.285029 24183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:24.290035 24192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.290012 24195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.290036 24193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.290303 24183 server_base.cc:1034] running on GCE node
I20260928 02:41:24.290454 24183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:24.290639 24183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:24.291767 24183 hybrid_clock.cc:648] HybridClock initialized: now 1790563284291756 us; error 30 us; skew 500 ppm
Sep 28 02:41:24 dist-test-slave-kkrh krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:24.294283 24183 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:41:24.294721 24183 webserver.cc:460] Webserver started at http://127.9.130.2:39667/ using document root <none> and password file <none>
I20260928 02:41:24.294948 24183 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:24.295002 24183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:24.296262 24183 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:24.296866 24202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.297039 24183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:24.297142 24183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "68b2054f0b8949de8d7a09b1ba8309b2"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "7984499ae22a0a0b034baa57a8de60e5"
server_key_iv: "8d0bc787deb44d4ef91978ef538980a6"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.297458 24183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:24.308238 24183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:24.309960 24183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:24.310109 24183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:24.310318 24183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:24.310706 24209 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 02:41:24.312076 24209 ts_tablet_manager.cc:1852] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 02:41:24.312340 24209 ts_tablet_manager.cc:1905] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:24.314484 24209 ts_tablet_manager.cc:1918] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:24.314574 24209 ts_tablet_manager.cc:1939] T 7a90b6711b874ee0a7c878971d9b1de0 P 68b2054f0b8949de8d7a09b1ba8309b2: Deleting consensus metadata
I20260928 02:41:24.314791 24183 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 02:41:24.314926 24183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 02:41:24.314975 24183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:24.315006 24183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.320667 24183 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:34443
I20260928 02:41:24.320760 24316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:34443 every 8 connection(s)
I20260928 02:41:24.321030 24183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:24.321705 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 24183
I20260928 02:41:24.321986 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:34727
--local_ip_for_outbound_sockets=127.9.130.3
--tserver_master_addrs=127.9.130.62:38647
--webserver_port=43839
--webserver_interface=127.9.130.3
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:41:24 dist-test-slave-kkrh krb5kdc[23413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:24.330135 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.321777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:52079 (local address 127.9.130.62:38647)
0928 02:41:24.321836 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.321839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.322092 (+ 253us) server_negotiation.cc:407] Connection header received
0928 02:41:24.322657 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.322659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.322676 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.322728 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:24.323327 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.323883 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.324883 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.325092 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.325175 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.327234 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:24.327235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:24.327235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:24.327243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:24.328734 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.329170 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.329171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.329171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.329230 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.329487 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.329488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.329489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.329603 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:24.329674 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.329954 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.330045 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:41:24.330303 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.321772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:52079)
0928 02:41:24.321954 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.322005 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.322537 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.322794 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.322798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.323048 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:24.323228 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.323231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.323973 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.323974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.324771 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.324773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.325184 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.325186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.325312 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.325814 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:24.325822 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:24.327138 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:24.328834 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.328841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.328843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.329080 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.329300 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.329301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.329301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.329410 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.329666 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:24.329672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:24.329808 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.329931 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.330079 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260928 02:41:24.330775 24317 heartbeater.cc:347] Connected to a master server at 127.9.130.62:38647
I20260928 02:41:24.330858 24317 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:24.331040 24317 heartbeater.cc:511] Master 127.9.130.62:38647 requested a full tablet report, sending...
I20260928 02:41:24.331416 23990 ts_manager.cc:194] Registered new tserver with Master: 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443)
I20260928 02:41:24.331928 23990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:52079
W20260928 02:41:24.402995 24320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:24.403193 24320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:24.403216 24320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:24.404505 24320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:24.404544 24320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:24.404572 24320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:41:24.406111 24320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:34727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=43839
--tserver_master_addrs=127.9.130.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:24.406359 24320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:24.406591 24320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:24.411538 24320 server_base.cc:1034] running on GCE node
W20260928 02:41:24.411509 24325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.411504 24328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.411515 24326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.412034 24320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:24.412298 24320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:24.413481 24320 hybrid_clock.cc:648] HybridClock initialized: now 1790563284413477 us; error 40 us; skew 500 ppm
Sep 28 02:41:24 dist-test-slave-kkrh krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:24.416061 24320 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:41:24.416532 24320 webserver.cc:460] Webserver started at http://127.9.130.3:43839/ using document root <none> and password file <none>
I20260928 02:41:24.416757 24320 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:24.416839 24320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:24.418241 24320 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:24.418928 24335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.419137 24320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:24.419261 24320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "9466be8e5d8a442c954713facc68b7e6"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "c64e09957cb066becf774a3f6bcbae61"
server_key_iv: "09fd882fe6590d734eb941995af858e4"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.419612 24320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:24.426515 24320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:24.428809 24320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:24.428963 24320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:24.429176 24320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:24.429483 24320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:24.429539 24320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.429579 24320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:24.429607 24320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.435580 24320 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:34727
I20260928 02:41:24.435616 24448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:34727 every 8 connection(s)
I20260928 02:41:24.436018 24320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:24.436662 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 24320
Sep 28 02:41:24 dist-test-slave-kkrh krb5kdc[23413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:24.444130 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.437055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.9.130.62:38647)
0928 02:41:24.437157 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.437159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.437169 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:41:24.437243 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.437244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.437255 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.437318 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:24.438024 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.438533 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.439584 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.439944 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.440013 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.441503 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:24.441504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:24.441504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:24.441511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:24.442893 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.443262 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.443262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.443262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.443324 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.443534 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.443535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.443536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.443645 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:24.443810 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.443906 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.444021 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
I20260928 02:41:24.446398 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.436787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:51401 (local address 127.9.130.62:38647)
0928 02:41:24.436841 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.436844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.437136 (+ 292us) server_negotiation.cc:407] Connection header received
0928 02:41:24.437649 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.437650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.437662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.437734 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:24.438738 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.439459 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.440634 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.440931 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.441012 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.443374 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:24.443375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:24.443375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:24.443386 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:24.444980 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.445430 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.445431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.445432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.445486 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.445745 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.445746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.445746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.445854 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:24.445927 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.446235 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.446308 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:41:24.446604 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.436816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:51401)
0928 02:41:24.436996 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.437044 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.437553 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.437824 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.437828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.438471 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:24.438655 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.438660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.439568 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.439569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.440553 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.440557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.441022 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.441022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.441118 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.441599 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:24.441608 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:24.443285 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:24.445053 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.445059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.445062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.445344 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.445561 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.445562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.445562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.445675 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.445921 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:24.445927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:24.446063 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.446201 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.446368 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260928 02:41:24.447072 24449 heartbeater.cc:347] Connected to a master server at 127.9.130.62:38647
I20260928 02:41:24.447176 24449 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:24.447351 24449 heartbeater.cc:511] Master 127.9.130.62:38647 requested a full tablet report, sending...
I20260928 02:41:24.447772 23990 ts_manager.cc:194] Registered new tserver with Master: 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727)
I20260928 02:41:24.448187 23990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:51401
I20260928 02:41:24.456049 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:41:24.458570 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.456360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.9.130.62:38647)
0928 02:41:24.456415 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.456417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.456480 (+ 63us) server_negotiation.cc:407] Connection header received
0928 02:41:24.456533 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.456533 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.456560 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.456616 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:24.456727 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.457174 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.457910 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.458087 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.458149 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.458284 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:24.458368 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:24.458428 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.458512 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.458526 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:41:24.458874 23990 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:24.460718 23990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:24.463230 24112 tablet_service.cc:1467] Processing CreateTablet for tablet e93a776381714762aa83b22ba915d9a8 (DEFAULT_TABLE table=test-table [id=52206071820046429f3fa9e7c19f3f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:24.463449 24112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93a776381714762aa83b22ba915d9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:24.465080 24070 tablet_bootstrap.cc:492] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e: Bootstrap starting.
I20260928 02:41:24.465410 24070 tablet_bootstrap.cc:654] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:24.466038 24070 tablet_bootstrap.cc:492] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e: No bootstrap required, opened a new log
I20260928 02:41:24.466097 24070 ts_tablet_manager.cc:1397] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:24.466449 24070 raft_consensus.cc:348] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:24.466540 24070 raft_consensus.cc:374] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:24.466598 24070 raft_consensus.cc:729] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2822f695eb1f4f06952efe8857ef957e, State: Initialized, Role: FOLLOWER
I20260928 02:41:24.466656 24070 consensus_queue.cc:260] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:24.466774 24070 ts_tablet_manager.cc:1428] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:24.467296 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.462788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:34443 (local address 127.0.0.1:42836)
0928 02:41:24.462969 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.462980 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.463088 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.463702 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.463703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.463704 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:24.463738 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.463739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.464753 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.464754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.466195 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.466197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.466989 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.466990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.467064 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.467085 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.467151 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.467214 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":113,"threads_started":1}
I20260928 02:41:24.467357 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.463119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.9.130.2:34443)
0928 02:41:24.463308 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.463312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.463372 (+ 60us) server_negotiation.cc:407] Connection header received
0928 02:41:24.463410 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.463413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.463429 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.463540 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:24.463868 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.464652 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.466257 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.466889 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.466957 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.467020 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.467177 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.467261 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":98,"threads_started":1}
I20260928 02:41:24.468201 24251 tablet_service.cc:1467] Processing CreateTablet for tablet e93a776381714762aa83b22ba915d9a8 (DEFAULT_TABLE table=test-table [id=52206071820046429f3fa9e7c19f3f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:24.468372 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.462732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:34727 (local address 127.0.0.1:35760)
0928 02:41:24.462783 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.462794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.462829 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.463512 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.463513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.463515 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:24.463556 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.463557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.465011 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.465012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.467168 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.467170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.468146 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.468148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.468244 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.468247 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.468295 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.468328 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1049}
I20260928 02:41:24.468466 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.462927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.9.130.3:34727)
0928 02:41:24.463344 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.463349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.463360 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:41:24.463399 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.463401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.463417 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.463510 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:24.463627 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.464897 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.467266 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.468046 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.468127 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.468183 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.468330 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.468370 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":81,"threads_started":1}
I20260928 02:41:24.468585 24251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93a776381714762aa83b22ba915d9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:24.469233 24383 tablet_service.cc:1467] Processing CreateTablet for tablet e93a776381714762aa83b22ba915d9a8 (DEFAULT_TABLE table=test-table [id=52206071820046429f3fa9e7c19f3f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:24.469535 24383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93a776381714762aa83b22ba915d9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:24.470451 24209 tablet_bootstrap.cc:492] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2: Bootstrap starting.
I20260928 02:41:24.471006 24209 tablet_bootstrap.cc:654] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:24.471421 24209 log.cc:824] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:24.472167 24209 tablet_bootstrap.cc:492] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2: No bootstrap required, opened a new log
I20260928 02:41:24.472246 24209 ts_tablet_manager.cc:1397] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:24.472257 24460 tablet_bootstrap.cc:492] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6: Bootstrap starting.
I20260928 02:41:24.472766 24460 tablet_bootstrap.cc:654] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:24.473196 24460 log.cc:824] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:24.473845 24460 tablet_bootstrap.cc:492] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6: No bootstrap required, opened a new log
I20260928 02:41:24.473896 24460 ts_tablet_manager.cc:1397] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:24.473851 24209 raft_consensus.cc:348] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:24.474050 24209 raft_consensus.cc:374] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:24.474102 24209 raft_consensus.cc:729] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b2054f0b8949de8d7a09b1ba8309b2, State: Initialized, Role: FOLLOWER
I20260928 02:41:24.474229 24209 consensus_queue.cc:260] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:24.474521 24209 ts_tablet_manager.cc:1428] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:24.474521 24317 heartbeater.cc:503] Master 127.9.130.62:38647 was elected leader, sending a full tablet report...
I20260928 02:41:24.475807 24460 raft_consensus.cc:348] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:24.476011 24460 raft_consensus.cc:374] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:24.476032 24460 raft_consensus.cc:729] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9466be8e5d8a442c954713facc68b7e6, State: Initialized, Role: FOLLOWER
I20260928 02:41:24.476133 24460 consensus_queue.cc:260] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:24.476347 24460 ts_tablet_manager.cc:1428] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:24.476414 24449 heartbeater.cc:503] Master 127.9.130.62:38647 was elected leader, sending a full tablet report...
I20260928 02:41:24.550686 24462 raft_consensus.cc:482] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:24.550812 24462 raft_consensus.cc:504] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:24.551173 24462 leader_election.cc:288] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727), 2822f695eb1f4f06952efe8857ef957e (127.9.130.1:46747)
I20260928 02:41:24.555310 24186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.551568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:60731 (local address 127.9.130.1:46747)
0928 02:41:24.551605 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.551607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.551771 (+ 164us) server_negotiation.cc:407] Connection header received
0928 02:41:24.551878 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.551879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.551907 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.551974 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:24.552108 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.552780 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.554278 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.554904 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.554977 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.555060 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.555195 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.555240 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 02:41:24.555310 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.551496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:37991 (local address 127.9.130.3:34727)
0928 02:41:24.551549 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.551551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.551562 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:41:24.551593 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.551594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.551607 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.551677 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:24.551868 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.552620 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.554278 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.554893 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.554977 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.554988 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.555195 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.555238 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:41:24.555344 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.551406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:34727 (local address 127.9.130.2:37991)
0928 02:41:24.551468 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.551479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.551524 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.551694 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.551696 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.551698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:24.551770 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.551772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.552746 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.552747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.554164 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.554167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.555003 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.555005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.555095 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.555097 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.555165 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.555220 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":51}
I20260928 02:41:24.555352 24464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.551480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46747 (local address 127.9.130.2:60731)
0928 02:41:24.551686 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.551695 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.551753 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.551987 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.551988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.551989 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:24.552037 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.552038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.552868 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.552869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.554164 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.554167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.555004 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.555005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.555093 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.555097 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.555165 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.555217 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":11,"thread_start_us":125,"threads_started":1}
I20260928 02:41:24.555735 24403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93a776381714762aa83b22ba915d9a8" candidate_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9466be8e5d8a442c954713facc68b7e6" is_pre_election: true
I20260928 02:41:24.555848 24132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93a776381714762aa83b22ba915d9a8" candidate_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2822f695eb1f4f06952efe8857ef957e" is_pre_election: true
I20260928 02:41:24.555948 24403 raft_consensus.cc:2454] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68b2054f0b8949de8d7a09b1ba8309b2 in term 0.
I20260928 02:41:24.556025 24132 raft_consensus.cc:2454] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68b2054f0b8949de8d7a09b1ba8309b2 in term 0.
I20260928 02:41:24.556207 24203 leader_election.cc:302] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68b2054f0b8949de8d7a09b1ba8309b2, 9466be8e5d8a442c954713facc68b7e6; no voters:
I20260928 02:41:24.556354 24462 raft_consensus.cc:2781] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:24.556414 24462 raft_consensus.cc:482] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:24.556452 24462 raft_consensus.cc:3037] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:24.557122 24462 raft_consensus.cc:504] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:24.557276 24462 leader_election.cc:288] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [CANDIDATE]: Term 1 election: Requested vote from peers 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727), 2822f695eb1f4f06952efe8857ef957e (127.9.130.1:46747)
I20260928 02:41:24.557436 24403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93a776381714762aa83b22ba915d9a8" candidate_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9466be8e5d8a442c954713facc68b7e6"
I20260928 02:41:24.557432 24132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93a776381714762aa83b22ba915d9a8" candidate_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2822f695eb1f4f06952efe8857ef957e"
I20260928 02:41:24.557510 24403 raft_consensus.cc:3037] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:24.557538 24132 raft_consensus.cc:3037] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:24.558243 24403 raft_consensus.cc:2454] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68b2054f0b8949de8d7a09b1ba8309b2 in term 1.
I20260928 02:41:24.558267 24132 raft_consensus.cc:2454] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68b2054f0b8949de8d7a09b1ba8309b2 in term 1.
I20260928 02:41:24.558456 24204 leader_election.cc:302] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2822f695eb1f4f06952efe8857ef957e, 68b2054f0b8949de8d7a09b1ba8309b2; no voters:
I20260928 02:41:24.558557 24462 raft_consensus.cc:2781] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:24.558698 24462 raft_consensus.cc:686] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 1 LEADER]: Becoming Leader. State: Replica: 68b2054f0b8949de8d7a09b1ba8309b2, State: Running, Role: LEADER
I20260928 02:41:24.558812 24462 consensus_queue.cc:237] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:24.559810 23990 catalog_manager.cc:5495] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2). New cstate: current_term: 1 leader_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } health_report { overall_health: UNKNOWN } } }
W20260928 02:41:24.571539 24318 tablet.cc:2369] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:24.587574 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.585189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.9.130.2:34443)
0928 02:41:24.585260 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.585263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.585273 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:41:24.585318 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.585319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.585338 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.585404 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:24.585600 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.586098 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.586829 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.587013 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.587067 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.587219 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:24.587314 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:24.587378 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.587418 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.587433 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:41:24.589408 24132 raft_consensus.cc:1261] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Refusing update from remote peer 68b2054f0b8949de8d7a09b1ba8309b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:24.589408 24403 raft_consensus.cc:1261] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6 [term 1 FOLLOWER]: Refusing update from remote peer 68b2054f0b8949de8d7a09b1ba8309b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:24.589809 24462 consensus_queue.cc:1035] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:24.589938 24465 consensus_queue.cc:1035] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:24.592290 24468 mvcc.cc:205] Tried to move back new op lower bound from 7334147213676146688 to 7334147213554274304. Current Snapshot: MvccSnapshot[applied={T|T < 7334147213676146688}]
I20260928 02:41:24.592321 24470 mvcc.cc:205] Tried to move back new op lower bound from 7334147213676146688 to 7334147213554274304. Current Snapshot: MvccSnapshot[applied={T|T < 7334147213676146688}]
I20260928 02:41:24.592790 24469 mvcc.cc:205] Tried to move back new op lower bound from 7334147213676146688 to 7334147213554274304. Current Snapshot: MvccSnapshot[applied={T|T < 7334147213676146688}]
I20260928 02:41:24.595156 23989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:24.595260 23989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:24.595978 23989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Sending DeleteTablet for 3 replicas of tablet e93a776381714762aa83b22ba915d9a8
I20260928 02:41:24.596292 24251 tablet_service.cc:1514] Processing DeleteTablet for tablet e93a776381714762aa83b22ba915d9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:24 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:42836
I20260928 02:41:24.596307 24112 tablet_service.cc:1514] Processing DeleteTablet for tablet e93a776381714762aa83b22ba915d9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:24 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:37038
I20260928 02:41:24.596328 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 24029
I20260928 02:41:24.596410 24187 tablet_replica.cc:330] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e: stopping tablet replica
I20260928 02:41:24.596478 24187 raft_consensus.cc:2229] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:24.596534 24187 raft_consensus.cc:2258] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:24.597489 24383 tablet_service.cc:1514] Processing DeleteTablet for tablet e93a776381714762aa83b22ba915d9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:24 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:35760
I20260928 02:41:24.597847 24477 tablet_replica.cc:330] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2: stopping tablet replica
I20260928 02:41:24.597946 24477 raft_consensus.cc:2229] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:24.598083 24477 raft_consensus.cc:2258] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:24.598242 24478 tablet_replica.cc:330] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6: stopping tablet replica
I20260928 02:41:24.598332 24478 raft_consensus.cc:2229] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:24.598428 24478 raft_consensus.cc:2258] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:24.598572 24477 ts_tablet_manager.cc:1905] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:24.598860 24478 ts_tablet_manager.cc:1905] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:24.600801 24477 ts_tablet_manager.cc:1918] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:24.600833 24478 ts_tablet_manager.cc:1918] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:24.600934 24478 log.cc:1196] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/e93a776381714762aa83b22ba915d9a8
I20260928 02:41:24.600956 24477 log.cc:1196] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/e93a776381714762aa83b22ba915d9a8
I20260928 02:41:24.601286 24478 ts_tablet_manager.cc:1939] T e93a776381714762aa83b22ba915d9a8 P 9466be8e5d8a442c954713facc68b7e6: Deleting consensus metadata
I20260928 02:41:24.601399 24477 ts_tablet_manager.cc:1939] T e93a776381714762aa83b22ba915d9a8 P 68b2054f0b8949de8d7a09b1ba8309b2: Deleting consensus metadata
I20260928 02:41:24.601693 23974 catalog_manager.cc:4841] TS 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727): tablet e93a776381714762aa83b22ba915d9a8 (table test-table [id=52206071820046429f3fa9e7c19f3f19]) successfully deleted
I20260928 02:41:24.601852 23975 catalog_manager.cc:4841] TS 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443): tablet e93a776381714762aa83b22ba915d9a8 (table test-table [id=52206071820046429f3fa9e7c19f3f19]) successfully deleted
W20260928 02:41:24.602293 23975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:46747 (error 108)
I20260928 02:41:24.602741 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.602593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46747 (local address 127.0.0.1:37042)
0928 02:41:24.602672 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.602711 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:46747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 02:41:24.602917 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 24183
W20260928 02:41:24.602885 23975 catalog_manager.cc:4584] TS 2822f695eb1f4f06952efe8857ef957e (127.9.130.1:46747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e93a776381714762aa83b22ba915d9a8: Network error: Client connection negotiation failed: client connection to 127.9.130.1:46747: connect: Connection refused (error 111)
I20260928 02:41:24.608196 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 24320
I20260928 02:41:24.612471 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 23957
I20260928 02:41:24.617265 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:38647
--webserver_interface=127.9.130.62
--webserver_port=46083
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:38647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:24.706703 24480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:24.706943 24480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:24.706983 24480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:24.708340 24480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:24.708407 24480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:24.708434 24480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:24.708465 24480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:24.708498 24480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:24.710188 24480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:38647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:38647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=46083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:24.710484 24480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:24.710731 24480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:24.715551 24486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.715579 24488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.715603 24485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.715898 24480 server_base.cc:1034] running on GCE node
I20260928 02:41:24.716127 24480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:24.716377 24480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:24.717536 24480 hybrid_clock.cc:648] HybridClock initialized: now 1790563284717514 us; error 44 us; skew 500 ppm
Sep 28 02:41:24 dist-test-slave-kkrh krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:24.720044 24480 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:41:24.720499 24480 webserver.cc:460] Webserver started at http://127.9.130.62:46083/ using document root <none> and password file <none>
I20260928 02:41:24.720744 24480 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:24.720813 24480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:24.722105 24480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:24.722805 24495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.722973 24480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:24.723069 24480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "5b2faf845e5249b387d7bc18099eb051"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "05b13f855ddccb90c071e7d7c8e3e229"
server_key_iv: "8d68fee9451ba5270b661b7aab10704a"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.723402 24480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:24.735327 24480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:24.737334 24480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:24.737489 24480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:24.741575 24480 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:38647
I20260928 02:41:24.741649 24547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:38647 every 8 connection(s)
I20260928 02:41:24.742010 24480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:24.743564 24548 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:41:24.744293 24548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Bootstrap starting.
I20260928 02:41:24.752161 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 24480
I20260928 02:41:24.752490 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:46747
--local_ip_for_outbound_sockets=127.9.130.1
--tserver_master_addrs=127.9.130.62:38647
--webserver_port=38461
--webserver_interface=127.9.130.1
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:41:24.752549 24548 log.cc:824] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:24.755009 24548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:41:24.755312 24548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051: Bootstrap complete.
I20260928 02:41:24.757593 24548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } }
I20260928 02:41:24.757812 24548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b2faf845e5249b387d7bc18099eb051, State: Initialized, Role: FOLLOWER
I20260928 02:41:24.757977 24548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } }
I20260928 02:41:24.758040 24548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:24.758067 24548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:24.758133 24548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:41:24.759104 24548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } }
I20260928 02:41:24.759195 24548 leader_election.cc:302] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b2faf845e5249b387d7bc18099eb051; no voters:
I20260928 02:41:24.759416 24548 leader_election.cc:288] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:41:24.759449 24552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:41:24.759557 24552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [term 3 LEADER]: Becoming Leader. State: Replica: 5b2faf845e5249b387d7bc18099eb051, State: Running, Role: LEADER
I20260928 02:41:24.759648 24552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } }
I20260928 02:41:24.759686 24548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:24.759945 24554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b2faf845e5249b387d7bc18099eb051. Latest consensus state: current_term: 3 leader_uuid: "5b2faf845e5249b387d7bc18099eb051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } } }
I20260928 02:41:24.760185 24554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:24.760294 24553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5b2faf845e5249b387d7bc18099eb051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2faf845e5249b387d7bc18099eb051" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38647 } } }
I20260928 02:41:24.760526 24562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:24.760560 24553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2faf845e5249b387d7bc18099eb051 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:24.761307 24562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:24.761420 24562 catalog_manager.cc:1259] Loaded cluster ID: f080871379f048b6a5e4decfa0cf97d2
I20260928 02:41:24.761663 24562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 02:41:24.761906 24562 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f03694559ad google::LogMessage::Fail() at ??:0
@ 0x7f036945888a google::LogMessage::SendToLog() at ??:0
@ 0x7f03694554de google::LogMessage::Flush() at ??:0
@ 0x7f0369456a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f03713760f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f037137f8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f036b3230d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f036b318954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f036c17c6db start_thread at ??:0
@ 0x7f0367f5361f clone at ??:0
W20260928 02:41:24.845299 24551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:24.845492 24551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:24.845516 24551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:24.846737 24551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:24.846779 24551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:24.846807 24551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:41:24.848312 24551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:46747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=38461
--tserver_master_addrs=127.9.130.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:24.848531 24551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:24.848776 24551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:24.854097 24587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.854115 24584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.854115 24585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.854242 24551 server_base.cc:1034] running on GCE node
I20260928 02:41:24.854521 24551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:24.854799 24551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:24.855968 24551 hybrid_clock.cc:648] HybridClock initialized: now 1790563284855945 us; error 46 us; skew 500 ppm
Sep 28 02:41:24 dist-test-slave-kkrh krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:24.858647 24551 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:41:24.859079 24551 webserver.cc:460] Webserver started at http://127.9.130.1:38461/ using document root <none> and password file <none>
I20260928 02:41:24.859308 24551 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:24.859377 24551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:24.860870 24551 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:24.861529 24594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.861719 24551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:41:24.861824 24551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "2822f695eb1f4f06952efe8857ef957e"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "eb87fc2517a056c207a3dd978eb71438"
server_key_iv: "383b9fabf1a47c5b7065a2ea9d34fb41"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.862207 24551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:24.872418 24551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:24.874677 24551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:24.874837 24551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:24.875074 24551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:24.875499 24601 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:41:24.877524 24551 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:41:24.877588 24551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:41:24.877633 24551 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:41:24.878260 24551 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:41:24.878315 24551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:41:24.878388 24601 tablet_bootstrap.cc:492] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e: Bootstrap starting.
I20260928 02:41:24.885908 24551 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:46747
I20260928 02:41:24.886051 24708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:46747 every 8 connection(s)
I20260928 02:41:24.886333 24551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:24.887745 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 24551
I20260928 02:41:24.888077 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:34443
--local_ip_for_outbound_sockets=127.9.130.2
--tserver_master_addrs=127.9.130.62:38647
--webserver_port=39667
--webserver_interface=127.9.130.2
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:41:24.889626 24601 log.cc:824] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:24.892546 24601 tablet_bootstrap.cc:492] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:41:24.892845 24601 tablet_bootstrap.cc:492] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e: Bootstrap complete.
I20260928 02:41:24.894048 24601 ts_tablet_manager.cc:1397] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260928 02:41:24.895526 24601 raft_consensus.cc:348] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:24.895876 24601 raft_consensus.cc:729] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2822f695eb1f4f06952efe8857ef957e, State: Initialized, Role: FOLLOWER
I20260928 02:41:24.896034 24601 consensus_queue.cc:260] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:24.896287 24601 ts_tablet_manager.cc:1428] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260928 02:41:24.958005 24711 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:41:24.886977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:55069)
0928 02:41:24.887142 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.887184 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.887804 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.957796 (+ 69992us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
W20260928 02:41:24.958223 24709 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:38647 (0 consecutive failures): Network error: Failed to ping master at 127.9.130.62:38647: Client connection negotiation failed: client connection to 127.9.130.62:38647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:41:24.958555 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.958432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:43321)
0928 02:41:24.958488 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.958527 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:41:24.958912 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.958827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:42869)
0928 02:41:24.958874 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.958896 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:41:24.959052 24709 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:41:24.971302 24712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:24.971501 24712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:24.971532 24712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:24.972772 24712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:24.972813 24712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:24.972839 24712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:41:24.974298 24712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:34443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=39667
--tserver_master_addrs=127.9.130.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:24.974546 24712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:24.974772 24712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:24.979708 24722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.979717 24720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.980006 24712 server_base.cc:1034] running on GCE node
W20260928 02:41:24.979779 24719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.980299 24712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:24.980540 24712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:24.981736 24712 hybrid_clock.cc:648] HybridClock initialized: now 1790563284981733 us; error 38 us; skew 500 ppm
Sep 28 02:41:24 dist-test-slave-kkrh krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:24.984730 24712 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:41:24.985337 24712 webserver.cc:460] Webserver started at http://127.9.130.2:39667/ using document root <none> and password file <none>
I20260928 02:41:24.985606 24712 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:24.985684 24712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:24.987196 24712 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:41:24.987807 24729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.988006 24712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:41:24.988128 24712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "68b2054f0b8949de8d7a09b1ba8309b2"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "7984499ae22a0a0b034baa57a8de60e5"
server_key_iv: "8d0bc787deb44d4ef91978ef538980a6"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.988507 24712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:25.005038 24712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:25.007318 24712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:25.007477 24712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:25.007704 24712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:25.008073 24712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:25.008138 24712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:25.008183 24712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:25.008227 24712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:25.014199 24712 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:34443
I20260928 02:41:25.014266 24842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:34443 every 8 connection(s)
I20260928 02:41:25.014638 24712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:25.015605 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 24712
I20260928 02:41:25.015769 24845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.015349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:51785)
0928 02:41:25.015521 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.015587 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260928 02:41:25.015925 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:34727
--local_ip_for_outbound_sockets=127.9.130.3
--tserver_master_addrs=127.9.130.62:38647
--webserver_port=43839
--webserver_interface=127.9.130.3
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:25.016292 24843 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:38647 (0 consecutive failures): Network error: Failed to ping master at 127.9.130.62:38647: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
I20260928 02:41:25.016621 24845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.016482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:40563)
0928 02:41:25.016574 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.016600 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 02:41:25.016924 24845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.016843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:37899)
0928 02:41:25.016885 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.016906 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:41:25.017120 24843 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:41:25.095571 24846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:25.095765 24846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:25.095788 24846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:25.096948 24846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:25.096988 24846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:25.097013 24846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:41:25.098361 24846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:34727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=43839
--tserver_master_addrs=127.9.130.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:25.098536 24846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:25.098694 24846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:25.103688 24846 server_base.cc:1034] running on GCE node
W20260928 02:41:25.103664 24852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:25.103672 24854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:25.103677 24851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:25.104043 24846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:25.104285 24846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:25.105443 24846 hybrid_clock.cc:648] HybridClock initialized: now 1790563285105420 us; error 54 us; skew 500 ppm
Sep 28 02:41:25 dist-test-slave-kkrh krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563285, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:25.107846 24846 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:41:25.108290 24846 webserver.cc:460] Webserver started at http://127.9.130.3:43839/ using document root <none> and password file <none>
I20260928 02:41:25.108525 24846 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:25.108592 24846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:25.109826 24846 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:25.110455 24861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:25.110618 24846 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:25.110711 24846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "9466be8e5d8a442c954713facc68b7e6"
format_stamp: "Formatted at 2026-09-28 02:41:23 on dist-test-slave-kkrh"
server_key: "c64e09957cb066becf774a3f6bcbae61"
server_key_iv: "09fd882fe6590d734eb941995af858e4"
server_key_version: "encryptionkey@0"
I20260928 02:41:25.111055 24846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:25.138258 24846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:25.140326 24846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:25.140491 24846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:25.140713 24846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:25.141027 24846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:25.141080 24846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:25.141114 24846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:25.141141 24846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:25.147099 24846 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:34727
I20260928 02:41:25.147145 24974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:34727 every 8 connection(s)
I20260928 02:41:25.147465 24846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:25.148594 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.148163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:33431)
0928 02:41:25.148317 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.148443 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
W20260928 02:41:25.149009 24975 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:38647 (0 consecutive failures): Network error: Failed to ping master at 127.9.130.62:38647: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
I20260928 02:41:25.149290 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.149200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:48965)
0928 02:41:25.149245 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.149272 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 02:41:25.149634 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.149555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:58011)
0928 02:41:25.149596 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.149619 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:41:25.149751 24975 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:41:25.150851 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 24846
I20260928 02:41:25.959681 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.959418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:53273)
0928 02:41:25.959567 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.959618 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260928 02:41:26.017612 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:26.017408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:51569)
0928 02:41:26.017522 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:26.017562 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":53,"threads_started":1}
I20260928 02:41:26.150239 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:26.150052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:50231)
0928 02:41:26.150162 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:26.150194 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260928 02:41:26.627481 24982 raft_consensus.cc:482] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:26.627660 24982 raft_consensus.cc:504] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:26.628046 24982 leader_election.cc:288] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727), 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443)
Sep 28 02:41:26 dist-test-slave-kkrh krb5kdc[23413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.2@KRBTEST.COM
Sep 28 02:41:26 dist-test-slave-kkrh krb5kdc[23413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.3@KRBTEST.COM
I20260928 02:41:26.639113 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:26.628220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:34443 (local address 127.9.130.1:56383)
0928 02:41:26.628354 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:26.628363 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:26.628418 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:26.629427 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:26.629428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:26.629778 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:26.629977 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:26.629983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:26.631075 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:26.631076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:26.632298 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:26.632301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:26.633114 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:26.633115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:26.633212 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:26.633739 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:26.633749 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:26.635515 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:26.637630 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:26.637636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:26.637638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:26.637853 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:26.638112 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:26.638113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:26.638113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:26.638221 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:26.638600 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:26.638607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:26.638739 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 02:41:26.638843 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:41:26.639022 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":16,"thread_start_us":73,"threads_started":1}
I20260928 02:41:26.639124 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:26.628360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:56383 (local address 127.9.130.2:34443)
0928 02:41:26.628502 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:26.628508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:26.628538 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:41:26.629077 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:26.629080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:26.629138 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:26.629296 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:26.630102 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:26.630971 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:26.632382 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:26.633025 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:26.633136 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:26.635630 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:26.635634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:26.635635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:26.635651 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:26.637527 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:26.637949 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:26.637952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:26.637952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:26.638042 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:26.638297 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:26.638298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:26.638298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:26.638527 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:26.638620 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:26.638893 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 02:41:26.639012 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
I20260928 02:41:26.639183 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:26.628295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:58029 (local address 127.9.130.3:34727)
0928 02:41:26.628457 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:26.628472 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:26.628508 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:41:26.629077 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:26.629080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:26.629138 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:26.629295 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:26.631123 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:26.632432 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:26.633346 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:26.633930 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:26.634038 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:26.636091 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:26.636094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:26.636095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:26.636103 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:26.637687 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:26.638053 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:26.638056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:26.638056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:26.638146 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:26.638353 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:26.638354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:26.638354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:26.638563 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:26.638688 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:26.638919 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 02:41:26.639067 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":51,"threads_started":1}
I20260928 02:41:26.639238 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:26.628197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:34727 (local address 127.9.130.1:58029)
0928 02:41:26.628346 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:26.628358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:26.628418 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:26.629360 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:26.629363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:26.629779 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:26.629978 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:26.629983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:26.632509 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:26.632510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:26.633260 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:26.633263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:26.634014 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:26.634015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:26.634055 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:26.634529 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:26.634535 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:26.635972 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:26.637795 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:26.637796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:26.637797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:26.637972 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:26.638224 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:26.638225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:26.638225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:26.638289 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:26.638669 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:26.638670 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:26.638739 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 02:41:26.638843 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:41:26.639110 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":123,"thread_start_us":88,"threads_started":1}
W20260928 02:41:26.640565 24596 leader_election.cc:341] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443): Not found: Tablet not found: e93a776381714762aa83b22ba915d9a8
W20260928 02:41:26.640707 24595 leader_election.cc:341] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727): Not found: Tablet not found: e93a776381714762aa83b22ba915d9a8
I20260928 02:41:26.640756 24595 leader_election.cc:302] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2822f695eb1f4f06952efe8857ef957e; no voters: 68b2054f0b8949de8d7a09b1ba8309b2, 9466be8e5d8a442c954713facc68b7e6
I20260928 02:41:26.640861 24982 raft_consensus.cc:2726] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:41:26.960336 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:26.960174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:35195)
0928 02:41:26.960255 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:26.960296 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 02:41:27.018288 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:27.018077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:41587)
0928 02:41:27.018207 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:27.018248 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260928 02:41:27.150902 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:27.150717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:34251)
0928 02:41:27.150823 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:27.150858 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260928 02:41:27.961035 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:27.960801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:49555)
0928 02:41:27.960946 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:27.960989 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260928 02:41:28.019105 24990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:28.018864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:50505)
0928 02:41:28.019020 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:28.019063 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260928 02:41:28.151563 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:28.151353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:55809)
0928 02:41:28.151489 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:28.151523 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260928 02:41:28.708348 24992 raft_consensus.cc:482] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:28.708431 24992 raft_consensus.cc:504] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:28.708630 24992 leader_election.cc:288] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727), 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443)
W20260928 02:41:28.709121 24595 leader_election.cc:341] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727): Not found: Tablet not found: e93a776381714762aa83b22ba915d9a8
W20260928 02:41:28.709223 24596 leader_election.cc:341] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443): Not found: Tablet not found: e93a776381714762aa83b22ba915d9a8
I20260928 02:41:28.709357 24596 leader_election.cc:302] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2822f695eb1f4f06952efe8857ef957e; no voters: 68b2054f0b8949de8d7a09b1ba8309b2, 9466be8e5d8a442c954713facc68b7e6
I20260928 02:41:28.709447 24992 raft_consensus.cc:2726] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:41:28.961793 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:28.961534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:49549)
0928 02:41:28.961696 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:28.961734 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260928 02:41:29.019793 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:29.019586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:46665)
0928 02:41:29.019711 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:29.019743 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260928 02:41:29.152238 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:29.152043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:34643)
0928 02:41:29.152154 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:29.152186 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260928 02:41:29.962565 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:29.962315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:52451)
0928 02:41:29.962477 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:29.962517 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260928 02:41:30.020521 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.020300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:50717)
0928 02:41:30.020436 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.020481 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260928 02:41:30.153035 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.152750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:33149)
0928 02:41:30.152952 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.152990 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
I20260928 02:41:30.777602 24999 raft_consensus.cc:482] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:30.777716 24999 raft_consensus.cc:504] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:30.777916 24999 leader_election.cc:288] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727), 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443)
W20260928 02:41:30.778332 24595 leader_election.cc:341] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727): Not found: Tablet not found: e93a776381714762aa83b22ba915d9a8
W20260928 02:41:30.778427 24596 leader_election.cc:341] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443): Not found: Tablet not found: e93a776381714762aa83b22ba915d9a8
I20260928 02:41:30.778501 24596 leader_election.cc:302] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2822f695eb1f4f06952efe8857ef957e; no voters: 68b2054f0b8949de8d7a09b1ba8309b2, 9466be8e5d8a442c954713facc68b7e6
I20260928 02:41:30.778589 24999 raft_consensus.cc:2726] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:41:30.963374 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.963143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:46667)
0928 02:41:30.963283 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.963329 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260928 02:41:31.021226 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:31.021017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:35043)
0928 02:41:31.021145 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:31.021186 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260928 02:41:31.153765 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:31.153553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:48319)
0928 02:41:31.153695 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:31.153728 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260928 02:41:31.964134 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:31.963906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:39875)
0928 02:41:31.964050 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:31.964087 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260928 02:41:32.021955 25004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:32.021714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:33209)
0928 02:41:32.021853 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:32.021902 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260928 02:41:32.154441 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:32.154239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:48515)
0928 02:41:32.154369 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:32.154403 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260928 02:41:32.964900 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:32.964623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:59533)
0928 02:41:32.964811 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:32.964851 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":60,"threads_started":1}
I20260928 02:41:33.022552 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:33.022363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:33769)
0928 02:41:33.022474 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:33.022507 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260928 02:41:33.155081 25008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:33.154883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:46101)
0928 02:41:33.155011 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:33.155041 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260928 02:41:33.965543 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:33.965337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:37493)
0928 02:41:33.965460 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:33.965497 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260928 02:41:34.023411 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:34.023104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:53425)
0928 02:41:34.023290 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:34.023347 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":117,"threads_started":1}
I20260928 02:41:34.155707 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:34.155476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:52881)
0928 02:41:34.155631 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:34.155667 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260928 02:41:34.495195 25012 raft_consensus.cc:482] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:34.495314 25012 raft_consensus.cc:504] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:34.495492 25012 leader_election.cc:288] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727), 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443)
W20260928 02:41:34.495853 24595 leader_election.cc:341] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727): Not found: Tablet not found: e93a776381714762aa83b22ba915d9a8
W20260928 02:41:34.495924 24596 leader_election.cc:341] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443): Not found: Tablet not found: e93a776381714762aa83b22ba915d9a8
I20260928 02:41:34.496021 24596 leader_election.cc:302] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2822f695eb1f4f06952efe8857ef957e; no voters: 68b2054f0b8949de8d7a09b1ba8309b2, 9466be8e5d8a442c954713facc68b7e6
I20260928 02:41:34.496109 25012 raft_consensus.cc:2726] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:41:34.966428 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:34.966187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:50283)
0928 02:41:34.966331 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:34.966381 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":104,"threads_started":1}
I20260928 02:41:35.024210 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:35.023956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:47269)
0928 02:41:35.024130 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:35.024168 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":96,"threads_started":1}
I20260928 02:41:35.156329 25015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:35.156143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:45713)
0928 02:41:35.156255 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:35.156287 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260928 02:41:35.967053 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:35.966849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:49491)
0928 02:41:35.966973 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:35.967008 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260928 02:41:36.024816 25017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:36.024623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:32787)
0928 02:41:36.024739 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:36.024772 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260928 02:41:36.157009 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:36.156743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:45783)
0928 02:41:36.156908 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:36.156944 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260928 02:41:36.967694 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:36.967488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:51119)
0928 02:41:36.967613 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:36.967650 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260928 02:41:37.025451 25020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:37.025239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:41355)
0928 02:41:37.025373 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:37.025409 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260928 02:41:37.157636 25021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:37.157446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:34113)
0928 02:41:37.157559 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:37.157593 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260928 02:41:37.968415 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:37.968205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:45549)
0928 02:41:37.968331 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:37.968369 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260928 02:41:38.026145 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:38.025907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:51449)
0928 02:41:38.026032 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:38.026098 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260928 02:41:38.158283 25024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:38.158082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:47711)
0928 02:41:38.158210 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:38.158240 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260928 02:41:38.360466 25025 raft_consensus.cc:482] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:38.360579 25025 raft_consensus.cc:504] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9466be8e5d8a442c954713facc68b7e6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34727 } } peers { permanent_uuid: "68b2054f0b8949de8d7a09b1ba8309b2" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 34443 } } peers { permanent_uuid: "2822f695eb1f4f06952efe8857ef957e" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46747 } }
I20260928 02:41:38.360790 25025 leader_election.cc:288] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727), 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443)
W20260928 02:41:38.361195 24595 leader_election.cc:341] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9466be8e5d8a442c954713facc68b7e6 (127.9.130.3:34727): Not found: Tablet not found: e93a776381714762aa83b22ba915d9a8
W20260928 02:41:38.361284 24596 leader_election.cc:341] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68b2054f0b8949de8d7a09b1ba8309b2 (127.9.130.2:34443): Not found: Tablet not found: e93a776381714762aa83b22ba915d9a8
I20260928 02:41:38.361366 24596 leader_election.cc:302] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2822f695eb1f4f06952efe8857ef957e; no voters: 68b2054f0b8949de8d7a09b1ba8309b2, 9466be8e5d8a442c954713facc68b7e6
I20260928 02:41:38.361446 25025 raft_consensus.cc:2726] T e93a776381714762aa83b22ba915d9a8 P 2822f695eb1f4f06952efe8857ef957e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:41:38.969101 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:38.968882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:55761)
0928 02:41:38.969019 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:38.969052 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260928 02:41:39.026881 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:39.026574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:58461)
0928 02:41:39.026743 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:39.026818 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":61,"threads_started":1}
I20260928 02:41:39.158886 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:39.158697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:53183)
0928 02:41:39.158814 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:39.158845 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260928 02:41:39.969733 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:39.969536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.1:50085)
0928 02:41:39.969653 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:39.969689 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
I20260928 02:41:40.027748 25030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:40.027482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.2:53975)
0928 02:41:40.027654 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:40.027693 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
I20260928 02:41:40.156809 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 24551
I20260928 02:41:40.159525 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:40.159336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:38647 (local address 127.9.130.3:48303)
0928 02:41:40.159455 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:40.159489 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
I20260928 02:41:40.161450 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 24712
I20260928 02:41:40.165643 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 24846
2026-09-28T02:41:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18328 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:40 dist-test-slave-kkrh krb5kdc[25034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:40 dist-test-slave-kkrh krb5kdc[25034](info): set up 2 sockets
Sep 28 02:41:40 dist-test-slave-kkrh krb5kdc[25034](info): commencing operation
krb5kdc: starting...
W20260928 02:41:41.195770 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:41:41 dist-test-slave-kkrh krb5kdc[25034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:41:41.310761 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:43253
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:43253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:41.388932 25047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:41.389123 25047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:41.389144 25047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:41.390283 25047 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:41:41.390331 25047 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:41:41.390344 25047 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:41:41.390355 25047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:41.390368 25047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:41.390419 25047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:41.390436 25047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:41.390458 25047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:41.391897 25047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:43253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:43253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:41.392145 25047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:41.392375 25047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:41.397574 25047 server_base.cc:1034] running on GCE node
W20260928 02:41:41.397552 25055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.397586 25053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.397600 25052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:41.398123 25047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:41.398360 25047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:41.399493 25047 hybrid_clock.cc:648] HybridClock initialized: now 1790563301399480 us; error 25 us; skew 500 ppm
Sep 28 02:41:41 dist-test-slave-kkrh krb5kdc[25034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:41.404867 25047 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:41:41.405403 25047 webserver.cc:460] Webserver started at http://127.9.130.62:43137/ using document root <none> and password file <none>
I20260928 02:41:41.405637 25047 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:41.405710 25047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:41.405875 25047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:41.406961 25047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "1c3e371363084369b5ca68b537126c86"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "74785a22ca266890d5fd5b254dc910ae"
server_key_iv: "062973ce5bb588e2264fd3e32fdcc430"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.407369 25047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "1c3e371363084369b5ca68b537126c86"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "74785a22ca266890d5fd5b254dc910ae"
server_key_iv: "062973ce5bb588e2264fd3e32fdcc430"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.408814 25047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:41:41.409596 25062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.409771 25047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:41:41.409878 25047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "1c3e371363084369b5ca68b537126c86"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "74785a22ca266890d5fd5b254dc910ae"
server_key_iv: "062973ce5bb588e2264fd3e32fdcc430"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.409970 25047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:41.433985 25047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:41.436393 25047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:41.436558 25047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:41.440759 25047 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:43253
I20260928 02:41:41.440791 25114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:43253 every 8 connection(s)
I20260928 02:41:41.441138 25047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:41.441963 25115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:41.444252 25115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Bootstrap starting.
I20260928 02:41:41.444766 25115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:41.445024 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 25047
I20260928 02:41:41.445034 25115 log.cc:824] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:41.445134 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:41:41.445394 9736 external_mini_cluster.cc:1428] Setting key 5e527008e00c42baffd7710f67e33a84
I20260928 02:41:41.445720 25115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: No bootstrap required, opened a new log
I20260928 02:41:41.447419 25115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } }
I20260928 02:41:41.447575 25115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:41.447619 25115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3e371363084369b5ca68b537126c86, State: Initialized, Role: FOLLOWER
I20260928 02:41:41.447755 25115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } }
I20260928 02:41:41.447875 25115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:41.447911 25115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:41.447955 25115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:41.448930 25115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } }
I20260928 02:41:41.449049 25115 leader_election.cc:302] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c3e371363084369b5ca68b537126c86; no voters:
I20260928 02:41:41.449223 25115 leader_election.cc:288] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:41.449252 25120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:41.449360 25120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 1 LEADER]: Becoming Leader. State: Replica: 1c3e371363084369b5ca68b537126c86, State: Running, Role: LEADER
I20260928 02:41:41.449476 25115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:41.449498 25120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } }
I20260928 02:41:41.449852 25120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3e371363084369b5ca68b537126c86. Latest consensus state: current_term: 1 leader_uuid: "1c3e371363084369b5ca68b537126c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } } }
I20260928 02:41:41.449867 25122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c3e371363084369b5ca68b537126c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } } }
I20260928 02:41:41.450054 25120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:41.450064 25122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:41.450665 25129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:41.451267 25129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:41.452824 25129 catalog_manager.cc:1347] Generated new cluster ID: 7e7d8ad83241492aa394b5f797ecb8bf
I20260928 02:41:41.452875 25129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:41:41 dist-test-slave-kkrh krb5kdc[25034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:41.458568 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.445901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.9.130.62:43253)
0928 02:41:41.446081 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:41.446096 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:41.446121 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:41:41.447448 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:41.447457 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:41.447520 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:41.447720 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:41.448507 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.449835 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.451096 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.451764 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.451918 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.454909 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:41.454912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:41.454913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:41.454923 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:41.456892 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.457317 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.457319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.457320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.457432 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.457661 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.457662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.457662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.457935 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:41.458071 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:41.458255 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 02:41:41.458380 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260928 02:41:41.470456 25129 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:41.470952 25129 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:41.476807 25129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Generated new TSK 0
I20260928 02:41:41.476971 25129 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:41:41.510761 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:43253
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:41.600307 25141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:41.600492 25141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:41.600514 25141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:41.601656 25141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:41.601703 25141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:41.601729 25141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:41:41.603119 25141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:41.603343 25141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:41.603579 25141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:41.609014 25147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.609027 25149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.609023 25146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:41.609072 25141 server_base.cc:1034] running on GCE node
I20260928 02:41:41.609404 25141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:41.609591 25141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:41.610728 25141 hybrid_clock.cc:648] HybridClock initialized: now 1790563301610716 us; error 29 us; skew 500 ppm
Sep 28 02:41:41 dist-test-slave-kkrh krb5kdc[25034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:41.613132 25141 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:41:41.613579 25141 webserver.cc:460] Webserver started at http://127.9.130.1:39625/ using document root <none> and password file <none>
I20260928 02:41:41.613749 25141 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:41.613787 25141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:41.613946 25141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:41.614917 25141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "efa3ceee4e04453f9f76ef6a3e72faab"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "65aeeb12a763cce0adfc4758dfa712fe"
server_key_iv: "04ad9d8a5a6e61065a6f0ca43f22be6f"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.615260 25141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "efa3ceee4e04453f9f76ef6a3e72faab"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "65aeeb12a763cce0adfc4758dfa712fe"
server_key_iv: "04ad9d8a5a6e61065a6f0ca43f22be6f"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.616631 25141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:41:41.617491 25156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.617717 25141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:41.617789 25141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "efa3ceee4e04453f9f76ef6a3e72faab"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "65aeeb12a763cce0adfc4758dfa712fe"
server_key_iv: "04ad9d8a5a6e61065a6f0ca43f22be6f"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.617882 25141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:41.635658 25141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:41.637696 25141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:41.637848 25141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:41.638098 25141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:41.638410 25141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:41.638465 25141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.638499 25141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:41.638525 25141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.644325 25141 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:38935
I20260928 02:41:41.644375 25269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:38935 every 8 connection(s)
I20260928 02:41:41.644672 25141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:41.645599 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 25141
I20260928 02:41:41.645697 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:41.645915 9736 external_mini_cluster.cc:1428] Setting key 4f84c1388d49e6ca87d66d72f58d38d4
Sep 28 02:41:41 dist-test-slave-kkrh krb5kdc[25034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
I20260928 02:41:41.655293 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.645357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:53205 (local address 127.9.130.62:43253)
0928 02:41:41.645445 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:41.645448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:41.645634 (+ 186us) server_negotiation.cc:407] Connection header received
0928 02:41:41.646373 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:41.646375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:41.646389 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:41.646449 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:41.647447 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.648173 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.649152 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.649543 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.649623 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.652320 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:41.652322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:41.652322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:41.652330 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:41.653713 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.654281 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.654282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.654282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.654343 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.654618 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.654619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.654619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.654781 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:41.654851 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:41.655155 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 02:41:41.655229 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:41:41.655431 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.645325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:53205)
0928 02:41:41.645514 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:41.645548 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:41.646202 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:41.646500 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:41.646503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:41.647139 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:41.647328 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.647333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.648268 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.648269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.649071 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.649073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.649655 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.649655 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.649757 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.650359 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:41.650367 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:41.652228 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:41.653807 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:41.653815 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:41.653817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:41.654133 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:41.654420 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:41.654421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:41.654421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:41.654550 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:41.654847 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:41.654853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:41.655037 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 02:41:41.655126 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:41:41.655242 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":79,"threads_started":1}
I20260928 02:41:41.655970 25270 heartbeater.cc:347] Connected to a master server at 127.9.130.62:43253
I20260928 02:41:41.656076 25270 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:41.656247 25270 heartbeater.cc:511] Master 127.9.130.62:43253 requested a full tablet report, sending...
I20260928 02:41:41.656827 25079 ts_manager.cc:194] Registered new tserver with Master: efa3ceee4e04453f9f76ef6a3e72faab (127.9.130.1:38935)
I20260928 02:41:41.657435 25079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:53205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:41:41.670866 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:43253
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:41.750615 25275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:41.750810 25275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:41.750833 25275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:41.752038 25275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:41.752079 25275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:41.752105 25275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:41:41.753522 25275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:41.753749 25275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:41.753980 25275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:41.758996 25275 server_base.cc:1034] running on GCE node
W20260928 02:41:41.758956 25281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.758968 25283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.758968 25280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:41.759393 25275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:41.759629 25275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:41.760774 25275 hybrid_clock.cc:648] HybridClock initialized: now 1790563301760742 us; error 56 us; skew 500 ppm
Sep 28 02:41:41 dist-test-slave-kkrh krb5kdc[25034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:41.763120 25275 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:41:41.763522 25275 webserver.cc:460] Webserver started at http://127.9.130.2:36563/ using document root <none> and password file <none>
I20260928 02:41:41.763741 25275 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:41.763808 25275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:41.763988 25275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:41.765020 25275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "0aac57c631024653aebf40e84862bff9"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "b1b66697727be14611a0187288422d76"
server_key_iv: "2d9bfb7f47f47dbc01620148f0bf7001"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.765354 25275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "0aac57c631024653aebf40e84862bff9"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "b1b66697727be14611a0187288422d76"
server_key_iv: "2d9bfb7f47f47dbc01620148f0bf7001"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.766615 25275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:41:41.767346 25290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.767551 25275 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:41:41.767654 25275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "0aac57c631024653aebf40e84862bff9"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "b1b66697727be14611a0187288422d76"
server_key_iv: "2d9bfb7f47f47dbc01620148f0bf7001"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.767736 25275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:41.793009 25275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:41.795907 25275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:41.796077 25275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:41.796319 25275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:41.796674 25275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:41.796721 25275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.796767 25275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:41.796801 25275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.802501 25275 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:33251
I20260928 02:41:41.802567 25403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:33251 every 8 connection(s)
I20260928 02:41:41.802870 25275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:41.805382 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 25275
I20260928 02:41:41.805495 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:41.805728 9736 external_mini_cluster.cc:1428] Setting key 9b9c4cbd5851cb6c3b8a3258a268075c
Sep 28 02:41:41 dist-test-slave-kkrh krb5kdc[25034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:41.812790 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.803614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:56797 (local address 127.9.130.62:43253)
0928 02:41:41.803675 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:41.803677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:41.803919 (+ 242us) server_negotiation.cc:407] Connection header received
0928 02:41:41.804489 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:41.804491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:41.804503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:41.804578 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:41.805671 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.806422 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.807428 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.807638 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.807713 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.809783 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:41.809785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:41.809785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:41.809793 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:41.811385 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.811773 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.811774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.811775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.811856 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.812136 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.812137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.812137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.812275 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:41.812347 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:41.812600 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 02:41:41.812683 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:41:41.812858 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.803600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:56797)
0928 02:41:41.803763 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:41.803796 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:41.804379 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:41.804653 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:41.804657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:41.805261 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:41.805554 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.805562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.806519 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.806521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.807319 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.807322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.807722 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.807723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.807905 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.808378 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:41.808386 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:41.809695 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:41.811473 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:41.811479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:41.811481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:41.811693 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:41.811932 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:41.811933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:41.811933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:41.812046 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:41.812348 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:41.812352 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:41.812462 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 02:41:41.812552 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:41:41.812669 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260928 02:41:41.813413 25404 heartbeater.cc:347] Connected to a master server at 127.9.130.62:43253
I20260928 02:41:41.813516 25404 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:41.813681 25404 heartbeater.cc:511] Master 127.9.130.62:43253 requested a full tablet report, sending...
I20260928 02:41:41.814118 25079 ts_manager.cc:194] Registered new tserver with Master: 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251)
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
I20260928 02:41:41.814533 25079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:56797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:41:41.831248 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:43253
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:41.910228 25409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:41.910418 25409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:41.910441 25409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:41.911607 25409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:41.911648 25409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:41.911674 25409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:41:41.913285 25409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:41:41.913518 25409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:41.913748 25409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:41.918809 25415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.918828 25417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.918828 25414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:41.918915 25409 server_base.cc:1034] running on GCE node
I20260928 02:41:41.919270 25409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:41.919507 25409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:41.920696 25409 hybrid_clock.cc:648] HybridClock initialized: now 1790563301920688 us; error 35 us; skew 500 ppm
Sep 28 02:41:41 dist-test-slave-kkrh krb5kdc[25034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:41.923192 25409 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:41:41.923647 25409 webserver.cc:460] Webserver started at http://127.9.130.3:40917/ using document root <none> and password file <none>
I20260928 02:41:41.923931 25409 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:41.924006 25409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:41.924145 25409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:41.925118 25409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "b826c1382ed445b9b9cc717e9ecee409"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "b9c7711022d251f8c6d1ca7058d62dc4"
server_key_iv: "9d08afcb9e17a63fa71296e95936d1d2"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.925470 25409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "b826c1382ed445b9b9cc717e9ecee409"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "b9c7711022d251f8c6d1ca7058d62dc4"
server_key_iv: "9d08afcb9e17a63fa71296e95936d1d2"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.926779 25409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:41:41.927527 25424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.927766 25409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:41.927912 25409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "b826c1382ed445b9b9cc717e9ecee409"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "b9c7711022d251f8c6d1ca7058d62dc4"
server_key_iv: "9d08afcb9e17a63fa71296e95936d1d2"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.928009 25409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:41.939200 25409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:41.941191 25409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:41.941363 25409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:41.941599 25409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:41.941926 25409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:41.941984 25409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.942028 25409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:41.942059 25409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.948371 25409 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:44485
I20260928 02:41:41.948419 25537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:44485 every 8 connection(s)
I20260928 02:41:41.948745 25409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:41:41 dist-test-slave-kkrh krb5kdc[25034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:41:41.955659 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 25409
I20260928 02:41:41.955802 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:41.956030 9736 external_mini_cluster.cc:1428] Setting key 93ed5b3a08f87bd2ecfbe05a72fc07ee
I20260928 02:41:41.958570 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.949489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:34929 (local address 127.9.130.62:43253)
0928 02:41:41.949565 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:41.949568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:41.949749 (+ 181us) server_negotiation.cc:407] Connection header received
0928 02:41:41.950301 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:41.950303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:41.950317 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:41.950400 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:41.951460 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.952096 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.953038 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.953248 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.953302 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.955480 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:41.955482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:41.955482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:41.955503 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:41.957289 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.957693 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.957694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.957695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.957764 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.958045 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.958046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.958047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.958154 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:41.958210 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:41.958436 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 02:41:41.958496 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:41:41.958819 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.949435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:34929)
0928 02:41:41.949604 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:41.949652 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:41.950183 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:41.950485 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:41.950488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:41.951196 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:41.951376 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.951381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.952180 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.952181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.952961 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.952963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.953320 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.953321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.953427 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.953929 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:41.953937 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:41.955374 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:41.957364 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:41.957370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:41.957371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:41.957604 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:41.957858 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:41.957859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:41.957860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:41.957969 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:41.958219 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:41.958223 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:41.958328 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 02:41:41.958410 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:41:41.958577 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
I20260928 02:41:41.959260 25538 heartbeater.cc:347] Connected to a master server at 127.9.130.62:43253
I20260928 02:41:41.959372 25538 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:41.959549 25538 heartbeater.cc:511] Master 127.9.130.62:43253 requested a full tablet report, sending...
I20260928 02:41:41.959931 25079 ts_manager.cc:194] Registered new tserver with Master: b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485)
I20260928 02:41:41.960307 25079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:34929
I20260928 02:41:41.967958 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:41:41.975905 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.969196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.9.130.62:43253)
0928 02:41:41.969319 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:41.969322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:41.969434 (+ 112us) server_negotiation.cc:407] Connection header received
0928 02:41:41.969559 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:41.969560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:41.969572 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:41.969634 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:41.970273 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.970768 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.971727 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.972014 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.972106 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.973394 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:41.973396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:41.973396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:41.973404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:41.974637 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.974996 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.974997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.974998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.975051 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.975285 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.975286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.975286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.975441 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:41.975542 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:41.975694 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:41:41.975772 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 02:41:41.977612 25079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:41.979103 25079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:41.985693 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.981466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.9.130.3:44485)
0928 02:41:41.981658 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:41.981663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:41.981910 (+ 247us) server_negotiation.cc:407] Connection header received
0928 02:41:41.982007 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:41.982010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:41.982027 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:41.982103 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:41.982300 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.983392 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.984711 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.985256 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.985353 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.985408 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:41.985568 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:41:41.985607 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":95,"threads_started":1}
I20260928 02:41:41.985716 25553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.981364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:44485 (local address 127.0.0.1:44324)
0928 02:41:41.981819 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:41.981850 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:41.981913 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:41.982177 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:41.982178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:41.982180 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:41.982239 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.982241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.983498 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.983499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.984590 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.984591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.985356 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.985357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.985436 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.985448 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:41:41.985550 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:41:41.985601 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260928 02:41:41.985792 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.981487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.9.130.1:38935)
0928 02:41:41.981658 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:41.981663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:41.981875 (+ 212us) server_negotiation.cc:407] Connection header received
0928 02:41:41.981951 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:41.981953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:41.981970 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:41.982044 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:41.982304 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.983442 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.984558 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.985280 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.985353 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.985408 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:41.985675 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 02:41:41.985718 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260928 02:41:41.985878 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.981688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.9.130.2:33251)
0928 02:41:41.981855 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:41.981859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:41.982136 (+ 277us) server_negotiation.cc:407] Connection header received
0928 02:41:41.982176 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:41.982178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:41.982192 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:41.982259 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:41.982456 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.983138 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.984624 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.985288 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.985353 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.985408 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:41.985757 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 02:41:41.985802 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260928 02:41:41.985958 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.981364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:38935 (local address 127.0.0.1:35630)
0928 02:41:41.981713 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:41.981758 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:41.981861 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:41.982149 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:41.982152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:41.982154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:41.982218 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.982220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.983545 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.983546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.984472 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.984474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.985445 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.985446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.985537 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.985540 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:41:41.985597 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:41:41.985628 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260928 02:41:41.985723 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.981566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:33251 (local address 127.0.0.1:51464)
0928 02:41:41.981973 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:41.982031 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:41.982080 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:41.982332 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:41.982332 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:41.982334 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:41.982361 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.982363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.983248 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.983249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.984529 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.984532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.985445 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.985446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.985531 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.985536 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:41:41.985587 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 02:41:41.985628 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":22,"thread_start_us":55,"threads_started":1}
I20260928 02:41:41.986654 25338 tablet_service.cc:1467] Processing CreateTablet for tablet 8fddf24b075d46b7bc4e2c3d4bc1d638 (DEFAULT_TABLE table=test-table [id=c4fd78455a214d3bac9581e91cd21652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:41.986706 25204 tablet_service.cc:1467] Processing CreateTablet for tablet 8fddf24b075d46b7bc4e2c3d4bc1d638 (DEFAULT_TABLE table=test-table [id=c4fd78455a214d3bac9581e91cd21652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:41.986917 25338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fddf24b075d46b7bc4e2c3d4bc1d638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:41.986948 25204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fddf24b075d46b7bc4e2c3d4bc1d638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:41.988658 25472 tablet_service.cc:1467] Processing CreateTablet for tablet 8fddf24b075d46b7bc4e2c3d4bc1d638 (DEFAULT_TABLE table=test-table [id=c4fd78455a214d3bac9581e91cd21652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:41.988943 25472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fddf24b075d46b7bc4e2c3d4bc1d638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:41.989591 25555 tablet_bootstrap.cc:492] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Bootstrap starting.
I20260928 02:41:41.989998 25556 tablet_bootstrap.cc:492] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9: Bootstrap starting.
I20260928 02:41:41.990146 25555 tablet_bootstrap.cc:654] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:41.990559 25555 log.cc:824] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:41.990623 25556 tablet_bootstrap.cc:654] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:41.991159 25555 tablet_bootstrap.cc:492] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: No bootstrap required, opened a new log
I20260928 02:41:41.991200 25555 ts_tablet_manager.cc:1397] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:41.991179 25556 log.cc:824] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:41.991984 25556 tablet_bootstrap.cc:492] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9: No bootstrap required, opened a new log
I20260928 02:41:41.992065 25556 ts_tablet_manager.cc:1397] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:41.993040 25555 raft_consensus.cc:348] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } }
I20260928 02:41:41.993194 25555 raft_consensus.cc:374] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:41.993240 25555 raft_consensus.cc:729] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa3ceee4e04453f9f76ef6a3e72faab, State: Initialized, Role: FOLLOWER
I20260928 02:41:41.993342 25555 consensus_queue.cc:260] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } }
I20260928 02:41:41.993618 25555 ts_tablet_manager.cc:1428] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 02:41:41.993680 25270 heartbeater.cc:503] Master 127.9.130.62:43253 was elected leader, sending a full tablet report...
I20260928 02:41:41.993686 25556 raft_consensus.cc:348] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } }
I20260928 02:41:41.993862 25556 raft_consensus.cc:374] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:41.993908 25556 raft_consensus.cc:729] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aac57c631024653aebf40e84862bff9, State: Initialized, Role: FOLLOWER
I20260928 02:41:41.994174 25556 consensus_queue.cc:260] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } }
I20260928 02:41:41.994444 25556 ts_tablet_manager.cc:1428] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:41.994463 25404 heartbeater.cc:503] Master 127.9.130.62:43253 was elected leader, sending a full tablet report...
I20260928 02:41:41.994627 25559 tablet_bootstrap.cc:492] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409: Bootstrap starting.
I20260928 02:41:41.995277 25559 tablet_bootstrap.cc:654] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:41.995724 25559 log.cc:824] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:41.996356 25559 tablet_bootstrap.cc:492] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409: No bootstrap required, opened a new log
I20260928 02:41:41.996404 25559 ts_tablet_manager.cc:1397] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:41.997800 25559 raft_consensus.cc:348] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } }
I20260928 02:41:41.997920 25559 raft_consensus.cc:374] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:41.997941 25559 raft_consensus.cc:729] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b826c1382ed445b9b9cc717e9ecee409, State: Initialized, Role: FOLLOWER
I20260928 02:41:41.998039 25559 consensus_queue.cc:260] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } }
I20260928 02:41:41.998351 25538 heartbeater.cc:503] Master 127.9.130.62:43253 was elected leader, sending a full tablet report...
I20260928 02:41:41.998337 25559 ts_tablet_manager.cc:1428] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:42.035597 25563 raft_consensus.cc:482] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:42.035713 25563 raft_consensus.cc:504] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } }
I20260928 02:41:42.036170 25563 leader_election.cc:288] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efa3ceee4e04453f9f76ef6a3e72faab (127.9.130.1:38935), 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251)
I20260928 02:41:42.040109 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:42.036370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:34107 (local address 127.9.130.1:38935)
0928 02:41:42.036423 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:42.036425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:42.036436 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:41:42.036532 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:42.036538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:42.036549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:42.036640 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:42.036839 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.037608 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.039145 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.039744 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.039840 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:42.039851 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:42.040002 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:41:42.040040 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:41:42.040148 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:42.036456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:56439 (local address 127.9.130.2:33251)
0928 02:41:42.036500 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:42.036502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:42.036580 (+ 78us) server_negotiation.cc:407] Connection header received
0928 02:41:42.036676 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:42.036678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:42.036689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:42.036741 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:42.036890 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.037617 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.039139 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.039744 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.039840 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:42.039851 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:42.040002 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:41:42.040040 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:41:42.040202 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:42.036325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:38935 (local address 127.9.130.3:34107)
0928 02:41:42.036377 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:42.036390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:42.036437 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:42.036665 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:42.036666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:42.036668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:42.036766 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:42.036768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.037700 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.037701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:42.039032 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:42.039034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.039843 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.039844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:42.039925 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:42.039927 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:41:42.039985 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:41:42.040034 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":104}
I20260928 02:41:42.040128 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:42.036340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:33251 (local address 127.9.130.3:56439)
0928 02:41:42.036502 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:42.036519 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:42.036571 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:42.036762 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:42.036762 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:42.036763 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:42.036820 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:42.036821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.037700 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.037701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:42.039032 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:42.039034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.039844 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.039844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:42.039923 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:42.039927 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:41:42.039985 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:41:42.040033 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":58,"thread_start_us":106,"threads_started":1}
I20260928 02:41:42.040540 25224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fddf24b075d46b7bc4e2c3d4bc1d638" candidate_uuid: "b826c1382ed445b9b9cc717e9ecee409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" is_pre_election: true
I20260928 02:41:42.040534 25358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fddf24b075d46b7bc4e2c3d4bc1d638" candidate_uuid: "b826c1382ed445b9b9cc717e9ecee409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aac57c631024653aebf40e84862bff9" is_pre_election: true
I20260928 02:41:42.040699 25358 raft_consensus.cc:2454] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b826c1382ed445b9b9cc717e9ecee409 in term 0.
I20260928 02:41:42.040699 25224 raft_consensus.cc:2454] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b826c1382ed445b9b9cc717e9ecee409 in term 0.
I20260928 02:41:42.040943 25426 leader_election.cc:302] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aac57c631024653aebf40e84862bff9, b826c1382ed445b9b9cc717e9ecee409; no voters:
I20260928 02:41:42.041090 25563 raft_consensus.cc:2781] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:42.041139 25563 raft_consensus.cc:482] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:42.041168 25563 raft_consensus.cc:3037] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:42.041854 25563 raft_consensus.cc:504] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } }
I20260928 02:41:42.041996 25563 leader_election.cc:288] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [CANDIDATE]: Term 1 election: Requested vote from peers efa3ceee4e04453f9f76ef6a3e72faab (127.9.130.1:38935), 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251)
I20260928 02:41:42.042232 25224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fddf24b075d46b7bc4e2c3d4bc1d638" candidate_uuid: "b826c1382ed445b9b9cc717e9ecee409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa3ceee4e04453f9f76ef6a3e72faab"
I20260928 02:41:42.042263 25358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fddf24b075d46b7bc4e2c3d4bc1d638" candidate_uuid: "b826c1382ed445b9b9cc717e9ecee409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aac57c631024653aebf40e84862bff9"
I20260928 02:41:42.042300 25224 raft_consensus.cc:3037] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:42.042371 25358 raft_consensus.cc:3037] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:42.043047 25224 raft_consensus.cc:2454] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b826c1382ed445b9b9cc717e9ecee409 in term 1.
I20260928 02:41:42.043169 25358 raft_consensus.cc:2454] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b826c1382ed445b9b9cc717e9ecee409 in term 1.
I20260928 02:41:42.043221 25427 leader_election.cc:302] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b826c1382ed445b9b9cc717e9ecee409, efa3ceee4e04453f9f76ef6a3e72faab; no voters:
I20260928 02:41:42.043308 25563 raft_consensus.cc:2781] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:42.043485 25563 raft_consensus.cc:686] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [term 1 LEADER]: Becoming Leader. State: Replica: b826c1382ed445b9b9cc717e9ecee409, State: Running, Role: LEADER
I20260928 02:41:42.043614 25563 consensus_queue.cc:237] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } }
I20260928 02:41:42.044489 25079 catalog_manager.cc:5495] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 reported cstate change: term changed from 0 to 1, leader changed from <none> to b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3). New cstate: current_term: 1 leader_uuid: "b826c1382ed445b9b9cc717e9ecee409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } health_report { overall_health: UNKNOWN } } }
W20260928 02:41:42.053375 25405 tablet.cc:2369] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:42.057314 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:42.055128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.9.130.3:44485)
0928 02:41:42.055183 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:42.055185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:42.055200 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:41:42.055226 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:42.055227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:42.055242 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:42.055301 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:42.055422 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.055922 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.056653 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.056832 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.056889 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:42.057011 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:42.057108 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:42.057169 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:42.057207 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 02:41:42.057223 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:41:42.058920 25224 raft_consensus.cc:1261] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Refusing update from remote peer b826c1382ed445b9b9cc717e9ecee409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:42.058920 25358 raft_consensus.cc:1261] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9 [term 1 FOLLOWER]: Refusing update from remote peer b826c1382ed445b9b9cc717e9ecee409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:42.059285 25565 consensus_queue.cc:1035] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:42.059396 25563 consensus_queue.cc:1035] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:42.061884 25568 mvcc.cc:205] Tried to move back new op lower bound from 7334147285231259648 to 7334147285171896320. Current Snapshot: MvccSnapshot[applied={T|T < 7334147285231259648}]
I20260928 02:41:42.061898 25570 mvcc.cc:205] Tried to move back new op lower bound from 7334147285231259648 to 7334147285171896320. Current Snapshot: MvccSnapshot[applied={T|T < 7334147285231259648}]
I20260928 02:41:42.062033 25569 mvcc.cc:205] Tried to move back new op lower bound from 7334147285231259648 to 7334147285171896320. Current Snapshot: MvccSnapshot[applied={T|T < 7334147285231259648}]
I20260928 02:41:42.064052 25079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:42.064138 25079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:42.064864 25079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Sending DeleteTablet for 3 replicas of tablet 8fddf24b075d46b7bc4e2c3d4bc1d638
I20260928 02:41:42.065186 25472 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fddf24b075d46b7bc4e2c3d4bc1d638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:42 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:44324
I20260928 02:41:42.065186 25338 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fddf24b075d46b7bc4e2c3d4bc1d638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:42 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:51464
I20260928 02:41:42.065218 25204 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fddf24b075d46b7bc4e2c3d4bc1d638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:42 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:35630
I20260928 02:41:42.065428 25578 tablet_replica.cc:330] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409: stopping tablet replica
I20260928 02:41:42.065438 25580 tablet_replica.cc:330] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: stopping tablet replica
I20260928 02:41:42.065474 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 25141
I20260928 02:41:42.065514 25580 raft_consensus.cc:2229] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:42.065514 25578 raft_consensus.cc:2229] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:42.065605 25580 raft_consensus.cc:2258] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:42.065619 25578 raft_consensus.cc:2258] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:42.065821 25579 tablet_replica.cc:330] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9: stopping tablet replica
I20260928 02:41:42.065897 25579 raft_consensus.cc:2229] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:42.065974 25579 raft_consensus.cc:2258] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:42.066315 25579 ts_tablet_manager.cc:1905] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:42.067027 25578 ts_tablet_manager.cc:1905] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:42.067966 25579 ts_tablet_manager.cc:1918] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:42.068027 25579 log.cc:1196] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/8fddf24b075d46b7bc4e2c3d4bc1d638
I20260928 02:41:42.068295 25579 ts_tablet_manager.cc:1939] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P 0aac57c631024653aebf40e84862bff9: Deleting consensus metadata
I20260928 02:41:42.068722 25578 ts_tablet_manager.cc:1918] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:42.068714 25064 catalog_manager.cc:4841] TS 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251): tablet 8fddf24b075d46b7bc4e2c3d4bc1d638 (table test-table [id=c4fd78455a214d3bac9581e91cd21652]) successfully deleted
I20260928 02:41:42.068780 25578 log.cc:1196] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/8fddf24b075d46b7bc4e2c3d4bc1d638
I20260928 02:41:42.069077 25578 ts_tablet_manager.cc:1939] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P b826c1382ed445b9b9cc717e9ecee409: Deleting consensus metadata
I20260928 02:41:42.069407 25064 catalog_manager.cc:4841] TS b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485): tablet 8fddf24b075d46b7bc4e2c3d4bc1d638 (table test-table [id=c4fd78455a214d3bac9581e91cd21652]) successfully deleted
W20260928 02:41:42.073268 25065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:38935 (error 108)
I20260928 02:41:42.073539 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 25275
I20260928 02:41:42.074185 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:42.073651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:38935 (local address 127.0.0.1:35646)
0928 02:41:42.074114 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:42.074147 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443}
W20260928 02:41:42.074906 25065 catalog_manager.cc:4584] TS efa3ceee4e04453f9f76ef6a3e72faab (127.9.130.1:38935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fddf24b075d46b7bc4e2c3d4bc1d638: Network error: Client connection negotiation failed: client connection to 127.9.130.1:38935: connect: Connection refused (error 111)
I20260928 02:41:42.080080 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 25409
I20260928 02:41:42.084714 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 25047
I20260928 02:42:02.090755 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:43253
--webserver_interface=127.9.130.62
--webserver_port=43137
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:43253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:02.170040 25582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:02.170236 25582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:02.170257 25582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:02.171386 25582 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:42:02.171427 25582 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:42:02.171439 25582 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:42:02.171451 25582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:02.171463 25582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:02.171476 25582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:02.171489 25582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:02.171536 25582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:02.172971 25582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:43253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:43253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=43137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:02.173224 25582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:02.173462 25582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:02.182494 25589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:02.182565 25582 server_base.cc:1034] running on GCE node
W20260928 02:42:02.182510 25588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:02.182577 25591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:02.182950 25582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:02.183248 25582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:02.184409 25582 hybrid_clock.cc:648] HybridClock initialized: now 1790563322184382 us; error 44 us; skew 500 ppm
Sep 28 02:42:02 dist-test-slave-kkrh krb5kdc[25034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:02.190294 25582 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:42:02.190801 25582 webserver.cc:460] Webserver started at http://127.9.130.62:43137/ using document root <none> and password file <none>
I20260928 02:42:02.191040 25582 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:02.191118 25582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:02.192605 25582 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:42:02.193316 25598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:02.193527 25582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:02.193629 25582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "1c3e371363084369b5ca68b537126c86"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "74785a22ca266890d5fd5b254dc910ae"
server_key_iv: "062973ce5bb588e2264fd3e32fdcc430"
server_key_version: "encryptionkey@0"
I20260928 02:42:02.194080 25582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:02.206736 25582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:02.208683 25582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:02.208847 25582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:02.212985 25582 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:43253
I20260928 02:42:02.213014 25650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:43253 every 8 connection(s)
I20260928 02:42:02.213413 25582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:02.214812 25651 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:42:02.215405 25651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Bootstrap starting.
I20260928 02:42:02.215865 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 25582
I20260928 02:42:02.216179 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:38935
--local_ip_for_outbound_sockets=127.9.130.1
--tserver_master_addrs=127.9.130.62:43253
--webserver_port=39625
--webserver_interface=127.9.130.1
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:42:02.224327 25651 log.cc:824] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:02.226269 25651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:42:02.226572 25651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Bootstrap complete.
I20260928 02:42:02.228875 25651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } }
I20260928 02:42:02.229097 25651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3e371363084369b5ca68b537126c86, State: Initialized, Role: FOLLOWER
I20260928 02:42:02.229264 25651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } }
I20260928 02:42:02.229321 25651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:02.229372 25651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:02.229418 25651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:42:02.230479 25651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } }
I20260928 02:42:02.230587 25651 leader_election.cc:302] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c3e371363084369b5ca68b537126c86; no voters:
I20260928 02:42:02.230780 25651 leader_election.cc:288] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:42:02.230814 25655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:42:02.230926 25655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 2 LEADER]: Becoming Leader. State: Replica: 1c3e371363084369b5ca68b537126c86, State: Running, Role: LEADER
I20260928 02:42:02.231010 25655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } }
I20260928 02:42:02.231040 25651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:02.231272 25655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3e371363084369b5ca68b537126c86. Latest consensus state: current_term: 2 leader_uuid: "1c3e371363084369b5ca68b537126c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } } }
I20260928 02:42:02.231335 25655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:02.231441 25656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1c3e371363084369b5ca68b537126c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } } }
I20260928 02:42:02.231535 25656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:02.231757 25659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:02.232594 25659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:02.232722 25659 catalog_manager.cc:1259] Loaded cluster ID: 7e7d8ad83241492aa394b5f797ecb8bf
I20260928 02:42:02.232966 25659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:02.233836 25659 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:02.234019 25659 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Loaded TSK: 0
I20260928 02:42:02.244750 25659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Generated new TSK 1
I20260928 02:42:02.244841 25659 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:42:02.310258 25653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:02.310456 25653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:02.310480 25653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:02.311722 25653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:02.311766 25653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:02.311794 25653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:42:02.313421 25653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:38935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=39625
--tserver_master_addrs=127.9.130.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:02.313661 25653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:02.313930 25653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:02.319722 25678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:02.319852 25679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:02.319861 25681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:02.320140 25653 server_base.cc:1034] running on GCE node
I20260928 02:42:02.320322 25653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:02.320559 25653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:02.321722 25653 hybrid_clock.cc:648] HybridClock initialized: now 1790563322321709 us; error 31 us; skew 500 ppm
Sep 28 02:42:02 dist-test-slave-kkrh krb5kdc[25034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:02.324420 25653 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:42:02.324925 25653 webserver.cc:460] Webserver started at http://127.9.130.1:39625/ using document root <none> and password file <none>
I20260928 02:42:02.325186 25653 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:02.325268 25653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:02.326977 25653 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:02.327713 25688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:02.327909 25653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:02.328001 25653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "efa3ceee4e04453f9f76ef6a3e72faab"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "65aeeb12a763cce0adfc4758dfa712fe"
server_key_iv: "04ad9d8a5a6e61065a6f0ca43f22be6f"
server_key_version: "encryptionkey@0"
I20260928 02:42:02.328392 25653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:02.340380 25653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:02.342820 25653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:02.342967 25653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:02.343210 25653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:02.343678 25695 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:42:02.345090 25653 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:42:02.345152 25653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:42:02.345183 25653 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:42:02.346031 25653 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:42:02.346083 25653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:42:02.346134 25695 tablet_bootstrap.cc:492] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Bootstrap starting.
I20260928 02:42:02.352983 25653 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:38935
I20260928 02:42:02.353389 25653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:42:02.356014 25802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:38935 every 8 connection(s)
I20260928 02:42:02.356546 25695 log.cc:824] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:02.358929 25695 tablet_bootstrap.cc:492] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:42:02.359216 25695 tablet_bootstrap.cc:492] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Bootstrap complete.
I20260928 02:42:02.360268 25695 ts_tablet_manager.cc:1397] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260928 02:42:02.361533 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 25653
I20260928 02:42:02.361358 25695 raft_consensus.cc:348] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } }
I20260928 02:42:02.361673 25695 raft_consensus.cc:729] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa3ceee4e04453f9f76ef6a3e72faab, State: Initialized, Role: FOLLOWER
I20260928 02:42:02.361812 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:33251
--local_ip_for_outbound_sockets=127.9.130.2
--tserver_master_addrs=127.9.130.62:43253
--webserver_port=36563
--webserver_interface=127.9.130.2
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:42:02.361814 25695 consensus_queue.cc:260] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } }
I20260928 02:42:02.362084 25695 ts_tablet_manager.cc:1428] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Sep 28 02:42:02 dist-test-slave-kkrh krb5kdc[25034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:02.369362 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.354253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:57095)
0928 02:42:02.355975 (+ 1722us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.356023 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:02.356806 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:02.357343 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:02.357347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:02.357710 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:02.357970 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.357977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.359676 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.359677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.361051 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.361054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.362344 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.362345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.362482 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.363063 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:02.363074 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:02.364843 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:02.367359 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:02.367365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:02.367367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:02.367667 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:02.368017 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:02.368018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:02.368018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:02.368148 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:02.368644 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:02.368650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:02.368816 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 02:42:02.368942 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 02:42:02.369145 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1674,"thread_start_us":98,"threads_started":1}
I20260928 02:42:02.369772 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.354373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:57095 (local address 127.9.130.62:43253)
0928 02:42:02.354564 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.354573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.356207 (+ 1634us) server_negotiation.cc:407] Connection header received
0928 02:42:02.357062 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.357069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.357135 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.357279 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:02.358110 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.359544 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.361153 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.362236 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.362367 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.364958 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:02.364962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:02.364962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:02.364974 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:02.367263 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:02.367771 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:02.367775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:02.367775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:02.367926 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:02.368236 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:02.368237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:02.368238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:02.368575 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:02.368674 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:02.369049 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 02:42:02.369199 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260928 02:42:02.370757 25803 heartbeater.cc:347] Connected to a master server at 127.9.130.62:43253
I20260928 02:42:02.370858 25803 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:02.371065 25803 heartbeater.cc:511] Master 127.9.130.62:43253 requested a full tablet report, sending...
I20260928 02:42:02.371938 25615 ts_manager.cc:194] Registered new tserver with Master: efa3ceee4e04453f9f76ef6a3e72faab (127.9.130.1:38935)
I20260928 02:42:02.373054 25615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:57095
I20260928 02:42:02.374001 25803 heartbeater.cc:503] Master 127.9.130.62:43253 was elected leader, sending a full tablet report...
I20260928 02:42:02.380090 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.372576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:38935 (local address 127.0.0.1:47834)
0928 02:42:02.375919 (+ 3343us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.375957 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:02.376056 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:02.376378 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:02.376381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:02.376383 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:02.376471 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.376473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.377666 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.377667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.378964 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.378967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.379763 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.379764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.379867 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.379881 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 02:42:02.379963 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:42:02.380014 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3306,"thread_start_us":89,"threads_started":1}
I20260928 02:42:02.380097 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.372821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.9.130.1:38935)
0928 02:42:02.373027 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.373032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.376018 (+ 2986us) server_negotiation.cc:407] Connection header received
0928 02:42:02.376154 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.376156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.376173 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.376362 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:02.376542 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.377595 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.379055 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.379670 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.379747 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.379797 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:02.379987 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 02:42:02.380031 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
I20260928 02:42:02.380789 25737 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fddf24b075d46b7bc4e2c3d4bc1d638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:42 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:47834
I20260928 02:42:02.381281 25812 tablet_replica.cc:330] stopping tablet replica
I20260928 02:42:02.381383 25812 raft_consensus.cc:2229] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:42:02.381448 25812 raft_consensus.cc:2258] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:02.381753 25812 ts_tablet_manager.cc:1905] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:02.384083 25812 ts_tablet_manager.cc:1918] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:02.384156 25812 log.cc:1196] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal/wals/8fddf24b075d46b7bc4e2c3d4bc1d638
I20260928 02:42:02.384369 25812 ts_tablet_manager.cc:1939] T 8fddf24b075d46b7bc4e2c3d4bc1d638 P efa3ceee4e04453f9f76ef6a3e72faab: Deleting consensus metadata
I20260928 02:42:02.384881 25601 catalog_manager.cc:4841] TS efa3ceee4e04453f9f76ef6a3e72faab (127.9.130.1:38935): tablet 8fddf24b075d46b7bc4e2c3d4bc1d638 (table test-table [id=c4fd78455a214d3bac9581e91cd21652]) successfully deleted
W20260928 02:42:02.446868 25809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:02.447060 25809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:02.447083 25809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:02.448767 25809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:02.448809 25809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:02.448836 25809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:42:02.450256 25809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:33251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=36563
--tserver_master_addrs=127.9.130.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:02.450492 25809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:02.450732 25809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:02.455638 25820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:02.455708 25809 server_base.cc:1034] running on GCE node
W20260928 02:42:02.455643 25817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:02.455637 25818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:02.456100 25809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:02.456377 25809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:02.457551 25809 hybrid_clock.cc:648] HybridClock initialized: now 1790563322457530 us; error 32 us; skew 500 ppm
Sep 28 02:42:02 dist-test-slave-kkrh krb5kdc[25034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:02.460098 25809 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:42:02.460455 25809 webserver.cc:460] Webserver started at http://127.9.130.2:36563/ using document root <none> and password file <none>
I20260928 02:42:02.460621 25809 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:02.460670 25809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:02.461906 25809 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:42:02.462424 25827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:02.462586 25809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:02.462667 25809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "0aac57c631024653aebf40e84862bff9"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "b1b66697727be14611a0187288422d76"
server_key_iv: "2d9bfb7f47f47dbc01620148f0bf7001"
server_key_version: "encryptionkey@0"
I20260928 02:42:02.463007 25809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:02.487200 25809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:02.490132 25809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:02.490276 25809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:02.490532 25809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:02.490924 25809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:02.490974 25809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:02.491014 25809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:02.491034 25809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:02.498133 25809 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:33251
I20260928 02:42:02.498188 25940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:33251 every 8 connection(s)
I20260928 02:42:02.498509 25809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:42:02 dist-test-slave-kkrh krb5kdc[25034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:02.507258 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 25809
I20260928 02:42:02.507607 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:44485
--local_ip_for_outbound_sockets=127.9.130.3
--tserver_master_addrs=127.9.130.62:43253
--webserver_port=40917
--webserver_interface=127.9.130.3
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:42:02.508608 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.499198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:59105 (local address 127.9.130.62:43253)
0928 02:42:02.499242 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.499244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.499461 (+ 217us) server_negotiation.cc:407] Connection header received
0928 02:42:02.500063 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.500065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.500082 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.500133 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:02.501080 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.501593 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.502629 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.502898 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.502989 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.505373 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:02.505380 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:02.505380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:02.505388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:02.506993 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:02.507491 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:02.507492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:02.507492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:02.507549 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:02.507900 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:02.507901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:02.507901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:02.508018 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:02.508114 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:02.508455 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 02:42:02.508538 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:42:02.508826 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.499175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:59105)
0928 02:42:02.499341 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.499376 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:02.499949 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:02.500197 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:02.500201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:02.500780 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:02.500977 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.500983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.501693 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.501694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.502517 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.502519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.502998 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.502999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.503126 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.503622 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:02.503630 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:02.505279 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:02.507096 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:02.507104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:02.507107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:02.507413 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:02.507623 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:02.507624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:02.507624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:02.507771 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:02.508091 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:02.508097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:02.508271 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 02:42:02.508376 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:42:02.508553 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260928 02:42:02.509444 25941 heartbeater.cc:347] Connected to a master server at 127.9.130.62:43253
I20260928 02:42:02.509543 25941 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:02.509733 25941 heartbeater.cc:511] Master 127.9.130.62:43253 requested a full tablet report, sending...
I20260928 02:42:02.510142 25615 ts_manager.cc:194] Registered new tserver with Master: 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251)
I20260928 02:42:02.510526 25615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:59105
W20260928 02:42:02.591686 25944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:02.591938 25944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:02.591976 25944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:02.593113 25944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:02.593183 25944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:02.593227 25944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:42:02.594653 25944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:44485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=40917
--tserver_master_addrs=127.9.130.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:02.594908 25944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:02.595140 25944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:02.600118 25952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:02.600172 25950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:02.600106 25949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:02.600389 25944 server_base.cc:1034] running on GCE node
I20260928 02:42:02.600574 25944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:02.600800 25944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:02.601969 25944 hybrid_clock.cc:648] HybridClock initialized: now 1790563322601944 us; error 35 us; skew 500 ppm
Sep 28 02:42:02 dist-test-slave-kkrh krb5kdc[25034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:02.604352 25944 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:42:02.604773 25944 webserver.cc:460] Webserver started at http://127.9.130.3:40917/ using document root <none> and password file <none>
I20260928 02:42:02.604995 25944 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:02.605083 25944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:02.606410 25944 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:02.607053 25959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:02.607231 25944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:02.607336 25944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "b826c1382ed445b9b9cc717e9ecee409"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "b9c7711022d251f8c6d1ca7058d62dc4"
server_key_iv: "9d08afcb9e17a63fa71296e95936d1d2"
server_key_version: "encryptionkey@0"
I20260928 02:42:02.607672 25944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:02.622992 25944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:02.624996 25944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:02.625150 25944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:02.625373 25944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:02.625663 25944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:02.625716 25944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:02.625756 25944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:02.625789 25944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:02.631518 25944 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:44485
I20260928 02:42:02.631582 26072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:44485 every 8 connection(s)
I20260928 02:42:02.632006 25944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:42:02.633169 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 25944
Sep 28 02:42:02 dist-test-slave-kkrh krb5kdc[25034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:02.640730 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.633589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.9.130.62:43253)
0928 02:42:02.633689 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.633691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.633808 (+ 117us) server_negotiation.cc:407] Connection header received
0928 02:42:02.633868 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.633868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.633898 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.633949 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:02.634701 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.635198 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.636066 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.636279 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.636352 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.637864 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:02.637865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:02.637866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:02.637871 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:02.639461 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:02.639845 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:02.639846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:02.639847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:02.639915 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:02.640123 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:02.640123 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:02.640124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:02.640237 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:02.640446 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:02.640484 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 02:42:02.640610 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":57,"threads_started":1}
I20260928 02:42:02.642115 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.632689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:58857 (local address 127.9.130.62:43253)
0928 02:42:02.632745 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.632748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.632981 (+ 233us) server_negotiation.cc:407] Connection header received
0928 02:42:02.633602 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.633603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.633618 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.633682 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:02.634918 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.635385 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.636417 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.636677 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.636743 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.639189 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:02.639190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:02.639191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:02.639197 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:02.640718 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:02.641302 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:02.641303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:02.641304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:02.641358 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:02.641626 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:02.641627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:02.641627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:02.641719 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:02.641769 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:02.642005 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 02:42:02.642061 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:42:02.642349 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.632702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:58857)
0928 02:42:02.632858 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.632903 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:02.633474 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:02.633820 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:02.633825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:02.634589 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:02.634830 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.634835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.635472 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.635473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.636330 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.636332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.636757 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.636775 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.636926 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.637311 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:02.637319 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:02.639098 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:02.640930 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:02.640936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:02.640938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:02.641232 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:02.641427 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:02.641427 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:02.641428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:02.641558 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:02.641782 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:02.641788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:02.641898 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 02:42:02.641981 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:42:02.642090 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":106,"threads_started":1}
I20260928 02:42:02.642755 26073 heartbeater.cc:347] Connected to a master server at 127.9.130.62:43253
I20260928 02:42:02.642863 26073 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:02.643061 26073 heartbeater.cc:511] Master 127.9.130.62:43253 requested a full tablet report, sending...
I20260928 02:42:02.643435 25615 ts_manager.cc:194] Registered new tserver with Master: b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485)
I20260928 02:42:02.643848 25615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:58857
I20260928 02:42:02.652256 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:42:02.654722 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.652575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.9.130.62:43253)
0928 02:42:02.652617 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.652619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.652687 (+ 68us) server_negotiation.cc:407] Connection header received
0928 02:42:02.652749 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.652749 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.652779 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.652835 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:42:02.652991 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.653431 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.654206 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.654382 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.654428 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.654540 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:42:02.654575 (+ 35us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 02:42:02.654679 (+ 104us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42926: token expired
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 02:42:02.654791 25806 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42926: token expired
W20260928 02:42:02.654922 25541 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.130.62:43253)
I20260928 02:42:02.660326 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.655183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.9.130.62:43253)
0928 02:42:02.655216 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.655217 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.655225 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:42:02.655269 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.655270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.655276 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.655324 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:02.655733 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.656202 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.656862 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.657031 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.657077 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.658205 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:02.658206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:02.658206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:02.658211 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:02.659389 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:02.659674 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:02.659674 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:02.659675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:02.659722 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:02.659934 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:02.659935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:02.659935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:02.660051 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:02.660127 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:02.660215 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 02:42:02.660270 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 02:42:02.661463 25615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:42:02.662993 25615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:42:02.665021 25737 tablet_service.cc:1467] Processing CreateTablet for tablet 3296f1d9c4be4a6ca45e7765bc298b54 (DEFAULT_TABLE table=test-table [id=98c6e804f83b40008a5e123ee77847c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:02.665230 25737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3296f1d9c4be4a6ca45e7765bc298b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:02.666700 25695 tablet_bootstrap.cc:492] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab: Bootstrap starting.
I20260928 02:42:02.667032 25695 tablet_bootstrap.cc:654] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:02.667593 25695 tablet_bootstrap.cc:492] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab: No bootstrap required, opened a new log
I20260928 02:42:02.667651 25695 ts_tablet_manager.cc:1397] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:42:02.667776 25695 raft_consensus.cc:348] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:02.667876 25695 raft_consensus.cc:374] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:02.667935 25695 raft_consensus.cc:729] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa3ceee4e04453f9f76ef6a3e72faab, State: Initialized, Role: FOLLOWER
I20260928 02:42:02.667979 25695 consensus_queue.cc:260] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:02.668068 25695 ts_tablet_manager.cc:1428] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:02.669024 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.664929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:33251 (local address 127.0.0.1:37786)
0928 02:42:02.665074 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.665082 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:02.665125 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:02.665369 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:02.665370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:02.665372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:02.665415 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.665417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.666440 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.666441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.667523 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.667527 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.668775 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.668776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.668850 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.668853 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:02.668918 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:42:02.668955 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260928 02:42:02.669080 26081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.665038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.9.130.2:33251)
0928 02:42:02.665210 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.665215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.665224 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:42:02.665268 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.665269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.665282 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.665378 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:02.665462 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.666340 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.667631 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.668684 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.668757 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.668798 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:02.668928 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 02:42:02.668957 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260928 02:42:02.669770 25863 tablet_service.cc:1467] Processing CreateTablet for tablet 3296f1d9c4be4a6ca45e7765bc298b54 (DEFAULT_TABLE table=test-table [id=98c6e804f83b40008a5e123ee77847c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:02.670116 25863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3296f1d9c4be4a6ca45e7765bc298b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:02.670400 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.664802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:44485 (local address 127.0.0.1:44014)
0928 02:42:02.664832 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.664842 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:02.664875 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:02.666941 (+ 2066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:02.666942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:02.666943 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:02.666982 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.666983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.668178 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.668179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.669213 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.669216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.670201 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.670203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.670266 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.670270 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:42:02.670303 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 02:42:02.670341 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 02:42:02.670715 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.664970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.9.130.3:44485)
0928 02:42:02.666796 (+ 1826us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.666801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.666812 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:42:02.666850 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.666852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.666868 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.666943 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:02.667027 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.668071 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.669345 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.670033 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.670514 (+ 481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.670575 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:02.670601 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 02:42:02.670640 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1790,"thread_start_us":84,"threads_started":1}
I20260928 02:42:02.671248 26007 tablet_service.cc:1467] Processing CreateTablet for tablet 3296f1d9c4be4a6ca45e7765bc298b54 (DEFAULT_TABLE table=test-table [id=98c6e804f83b40008a5e123ee77847c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:02.671543 26007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3296f1d9c4be4a6ca45e7765bc298b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:02.673147 26083 tablet_bootstrap.cc:492] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9: Bootstrap starting.
I20260928 02:42:02.673684 26083 tablet_bootstrap.cc:654] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:02.674211 26083 log.cc:824] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:02.674304 26084 tablet_bootstrap.cc:492] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409: Bootstrap starting.
I20260928 02:42:02.674942 26084 tablet_bootstrap.cc:654] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:02.675055 26083 tablet_bootstrap.cc:492] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9: No bootstrap required, opened a new log
I20260928 02:42:02.675122 26083 ts_tablet_manager.cc:1397] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:42:02.675341 26084 log.cc:824] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:02.676088 26084 tablet_bootstrap.cc:492] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409: No bootstrap required, opened a new log
I20260928 02:42:02.676142 26084 ts_tablet_manager.cc:1397] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:42:02.676892 26083 raft_consensus.cc:348] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:02.677111 26083 raft_consensus.cc:374] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:02.677162 26083 raft_consensus.cc:729] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aac57c631024653aebf40e84862bff9, State: Initialized, Role: FOLLOWER
I20260928 02:42:02.677281 26083 consensus_queue.cc:260] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:02.677552 26083 ts_tablet_manager.cc:1428] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:42:02.677582 25941 heartbeater.cc:503] Master 127.9.130.62:43253 was elected leader, sending a full tablet report...
I20260928 02:42:02.677743 26084 raft_consensus.cc:348] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:02.677897 26084 raft_consensus.cc:374] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:02.677942 26084 raft_consensus.cc:729] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b826c1382ed445b9b9cc717e9ecee409, State: Initialized, Role: FOLLOWER
I20260928 02:42:02.678057 26084 consensus_queue.cc:260] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:02.678324 26084 ts_tablet_manager.cc:1428] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:42:02.678370 26073 heartbeater.cc:503] Master 127.9.130.62:43253 was elected leader, sending a full tablet report...
I20260928 02:42:02.679230 26088 raft_consensus.cc:482] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:42:02.679315 26088 raft_consensus.cc:504] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:02.679586 26088 leader_election.cc:288] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251), efa3ceee4e04453f9f76ef6a3e72faab (127.9.130.1:38935)
I20260928 02:42:02.682313 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.679711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:33251 (local address 127.9.130.3:49785)
0928 02:42:02.679756 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.679765 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:02.679805 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:02.679996 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:02.679997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:02.679999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:02.680067 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.680068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.680689 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.680689 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.681444 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.681447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.682063 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.682063 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.682149 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.682152 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:02.682208 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:42:02.682251 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:42:02.682374 26081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.679751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:49785 (local address 127.9.130.2:33251)
0928 02:42:02.679793 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.679796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.679805 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:42:02.679923 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.679924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.679937 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.679990 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:02.680134 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.680619 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.681526 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.681954 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.682019 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.682028 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:02.682277 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 02:42:02.682317 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:42:02.682391 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.679913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:58431 (local address 127.9.130.1:38935)
0928 02:42:02.679949 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.679951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.679958 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:42:02.680020 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.680021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.680032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.680084 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:02.680239 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.680736 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.681631 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.682060 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.682126 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.682135 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:02.682276 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 02:42:02.682322 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:42:02.682595 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.679735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:38935 (local address 127.9.130.3:58431)
0928 02:42:02.679904 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.679914 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:02.679955 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:02.680091 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:02.680091 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:02.680092 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:02.680171 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.680172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.680816 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.680816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.681545 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.681547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.682127 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.682128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.682188 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.682191 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:02.682490 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 02:42:02.682528 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":108,"threads_started":1}
I20260928 02:42:02.682715 25895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296f1d9c4be4a6ca45e7765bc298b54" candidate_uuid: "b826c1382ed445b9b9cc717e9ecee409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aac57c631024653aebf40e84862bff9" is_pre_election: true
I20260928 02:42:02.682827 25895 raft_consensus.cc:2454] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b826c1382ed445b9b9cc717e9ecee409 in term 0.
I20260928 02:42:02.682878 25757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296f1d9c4be4a6ca45e7765bc298b54" candidate_uuid: "b826c1382ed445b9b9cc717e9ecee409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" is_pre_election: true
I20260928 02:42:02.683019 25757 raft_consensus.cc:2454] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b826c1382ed445b9b9cc717e9ecee409 in term 0.
I20260928 02:42:02.683017 25961 leader_election.cc:302] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aac57c631024653aebf40e84862bff9, b826c1382ed445b9b9cc717e9ecee409; no voters:
I20260928 02:42:02.683185 26088 raft_consensus.cc:2781] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:42:02.683251 26088 raft_consensus.cc:482] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:42:02.683284 26088 raft_consensus.cc:3037] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:02.683897 26088 raft_consensus.cc:504] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:02.684036 26088 leader_election.cc:288] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [CANDIDATE]: Term 1 election: Requested vote from peers 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251), efa3ceee4e04453f9f76ef6a3e72faab (127.9.130.1:38935)
I20260928 02:42:02.684170 25757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296f1d9c4be4a6ca45e7765bc298b54" candidate_uuid: "b826c1382ed445b9b9cc717e9ecee409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa3ceee4e04453f9f76ef6a3e72faab"
I20260928 02:42:02.684183 25895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296f1d9c4be4a6ca45e7765bc298b54" candidate_uuid: "b826c1382ed445b9b9cc717e9ecee409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aac57c631024653aebf40e84862bff9"
I20260928 02:42:02.684247 25895 raft_consensus.cc:3037] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:02.684238 25757 raft_consensus.cc:3037] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:02.685052 25895 raft_consensus.cc:2454] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b826c1382ed445b9b9cc717e9ecee409 in term 1.
I20260928 02:42:02.685230 25961 leader_election.cc:302] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aac57c631024653aebf40e84862bff9, b826c1382ed445b9b9cc717e9ecee409; no voters:
I20260928 02:42:02.685330 26088 raft_consensus.cc:2781] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:02.685338 25757 raft_consensus.cc:2454] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b826c1382ed445b9b9cc717e9ecee409 in term 1.
I20260928 02:42:02.685513 26088 raft_consensus.cc:686] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [term 1 LEADER]: Becoming Leader. State: Replica: b826c1382ed445b9b9cc717e9ecee409, State: Running, Role: LEADER
I20260928 02:42:02.685609 26088 consensus_queue.cc:237] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:02.686345 25615 catalog_manager.cc:5495] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 reported cstate change: term changed from 0 to 1, leader changed from <none> to b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3). New cstate: current_term: 1 leader_uuid: "b826c1382ed445b9b9cc717e9ecee409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } health_report { overall_health: UNKNOWN } } }
I20260928 02:42:02.691172 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.688954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.9.130.3:44485)
0928 02:42:02.689012 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.689014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.689022 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:42:02.689100 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.689100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.689116 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.689171 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:42:02.689293 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.689733 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.690437 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.690635 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.690717 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.690871 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:42:02.690958 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:42:02.691047 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:02.691069 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 02:42:02.691086 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:42:02.692803 25757 raft_consensus.cc:1261] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Refusing update from remote peer b826c1382ed445b9b9cc717e9ecee409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:02.692808 25895 raft_consensus.cc:1261] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9 [term 1 FOLLOWER]: Refusing update from remote peer b826c1382ed445b9b9cc717e9ecee409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:02.693106 26090 consensus_queue.cc:1035] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:02.693246 26088 consensus_queue.cc:1035] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:02.695423 26094 mvcc.cc:205] Tried to move back new op lower bound from 7334147369747365888 to 7334147369721102336. Current Snapshot: MvccSnapshot[applied={T|T < 7334147369747365888}]
I20260928 02:42:02.696106 26092 mvcc.cc:205] Tried to move back new op lower bound from 7334147369747365888 to 7334147369721102336. Current Snapshot: MvccSnapshot[applied={T|T < 7334147369747365888}]
I20260928 02:42:02.698094 25615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:02.698168 25615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:02.698804 25615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Sending DeleteTablet for 3 replicas of tablet 3296f1d9c4be4a6ca45e7765bc298b54
I20260928 02:42:02.699069 26007 tablet_service.cc:1514] Processing DeleteTablet for tablet 3296f1d9c4be4a6ca45e7765bc298b54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:02 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:44014
I20260928 02:42:02.699091 25737 tablet_service.cc:1514] Processing DeleteTablet for tablet 3296f1d9c4be4a6ca45e7765bc298b54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:02 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:47834
I20260928 02:42:02.699137 25863 tablet_service.cc:1514] Processing DeleteTablet for tablet 3296f1d9c4be4a6ca45e7765bc298b54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:02 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:37786
I20260928 02:42:02.699182 25812 tablet_replica.cc:330] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab: stopping tablet replica
I20260928 02:42:02.699234 25812 raft_consensus.cc:2229] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:42:02.699291 25812 raft_consensus.cc:2258] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:02.699450 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 25653
I20260928 02:42:02.699613 25812 ts_tablet_manager.cc:1905] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:02.699458 26103 tablet_replica.cc:330] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9: stopping tablet replica
I20260928 02:42:02.701247 26103 raft_consensus.cc:2229] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:42:02.701334 26103 raft_consensus.cc:2258] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:02.701738 26103 ts_tablet_manager.cc:1905] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:02.703274 26103 ts_tablet_manager.cc:1918] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:02.703365 26103 log.cc:1196] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/3296f1d9c4be4a6ca45e7765bc298b54
I20260928 02:42:02.703637 26103 ts_tablet_manager.cc:1939] T 3296f1d9c4be4a6ca45e7765bc298b54 P 0aac57c631024653aebf40e84862bff9: Deleting consensus metadata
I20260928 02:42:02.703944 26102 tablet_replica.cc:330] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409: stopping tablet replica
I20260928 02:42:02.704101 26102 raft_consensus.cc:2229] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:42:02.704123 25600 catalog_manager.cc:4841] TS 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251): tablet 3296f1d9c4be4a6ca45e7765bc298b54 (table test-table [id=98c6e804f83b40008a5e123ee77847c9]) successfully deleted
I20260928 02:42:02.704310 26102 raft_consensus.cc:2258] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:02.704797 26102 ts_tablet_manager.cc:1905] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:02.706156 26102 ts_tablet_manager.cc:1918] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:02.706243 26102 log.cc:1196] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/3296f1d9c4be4a6ca45e7765bc298b54
I20260928 02:42:02.706508 26102 ts_tablet_manager.cc:1939] T 3296f1d9c4be4a6ca45e7765bc298b54 P b826c1382ed445b9b9cc717e9ecee409: Deleting consensus metadata
W20260928 02:42:02.706688 25601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:38935 (error 108)
I20260928 02:42:02.706943 25600 catalog_manager.cc:4841] TS b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485): tablet 3296f1d9c4be4a6ca45e7765bc298b54 (table test-table [id=98c6e804f83b40008a5e123ee77847c9]) successfully deleted
I20260928 02:42:02.707072 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 25809
I20260928 02:42:02.707651 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.707257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:38935 (local address 127.0.0.1:47846)
0928 02:42:02.707573 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.707613 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297}
W20260928 02:42:02.708801 25601 catalog_manager.cc:4584] TS efa3ceee4e04453f9f76ef6a3e72faab (127.9.130.1:38935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3296f1d9c4be4a6ca45e7765bc298b54: Network error: Client connection negotiation failed: client connection to 127.9.130.1:38935: connect: Connection refused (error 111)
I20260928 02:42:02.712935 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 25944
I20260928 02:42:02.717784 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 25582
I20260928 02:42:22.723135 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:43253
--webserver_interface=127.9.130.62
--webserver_port=43137
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:43253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:22.803316 26105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:22.803522 26105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:22.803544 26105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:22.804760 26105 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:42:22.804800 26105 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:42:22.804814 26105 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:42:22.804826 26105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:22.804838 26105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:22.804852 26105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:22.804903 26105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:22.804921 26105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:22.806381 26105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:43253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:43253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=43137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:22.806627 26105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:22.806866 26105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:22.812067 26110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:22.812119 26105 server_base.cc:1034] running on GCE node
W20260928 02:42:22.812232 26111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:22.812135 26113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:22.812557 26105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:22.812857 26105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:22.814015 26105 hybrid_clock.cc:648] HybridClock initialized: now 1790563342814006 us; error 45 us; skew 500 ppm
Sep 28 02:42:22 dist-test-slave-kkrh krb5kdc[25034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:22.821292 26105 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:42:22.821770 26105 webserver.cc:460] Webserver started at http://127.9.130.62:43137/ using document root <none> and password file <none>
I20260928 02:42:22.822005 26105 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:22.822073 26105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:22.823536 26105 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:42:22.824420 26120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:22.824620 26105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:42:22.824726 26105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "1c3e371363084369b5ca68b537126c86"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "74785a22ca266890d5fd5b254dc910ae"
server_key_iv: "062973ce5bb588e2264fd3e32fdcc430"
server_key_version: "encryptionkey@0"
I20260928 02:42:22.825107 26105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:22.840953 26105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:22.843322 26105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:22.843485 26105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:22.847741 26105 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:43253
I20260928 02:42:22.847776 26172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:43253 every 8 connection(s)
I20260928 02:42:22.848213 26105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:22.849684 26173 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:42:22.850270 26173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Bootstrap starting.
I20260928 02:42:22.858321 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 26105
I20260928 02:42:22.858668 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:38935
--local_ip_for_outbound_sockets=127.9.130.1
--tserver_master_addrs=127.9.130.62:43253
--webserver_port=39625
--webserver_interface=127.9.130.1
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:42:22.858935 26173 log.cc:824] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:22.861405 26173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:42:22.861716 26173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Bootstrap complete.
I20260928 02:42:22.864063 26173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } }
I20260928 02:42:22.864279 26173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3e371363084369b5ca68b537126c86, State: Initialized, Role: FOLLOWER
I20260928 02:42:22.864452 26173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } }
I20260928 02:42:22.864514 26173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:22.864573 26173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:22.864620 26173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:42:22.865598 26173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } }
I20260928 02:42:22.865680 26173 leader_election.cc:302] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c3e371363084369b5ca68b537126c86; no voters:
I20260928 02:42:22.865873 26173 leader_election.cc:288] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:42:22.865914 26177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:42:22.866079 26177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [term 3 LEADER]: Becoming Leader. State: Replica: 1c3e371363084369b5ca68b537126c86, State: Running, Role: LEADER
I20260928 02:42:22.866142 26173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:22.866251 26177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } }
I20260928 02:42:22.866523 26177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3e371363084369b5ca68b537126c86. Latest consensus state: current_term: 3 leader_uuid: "1c3e371363084369b5ca68b537126c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } } }
I20260928 02:42:22.866611 26177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:22.866552 26179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1c3e371363084369b5ca68b537126c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e371363084369b5ca68b537126c86" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 43253 } } }
I20260928 02:42:22.866811 26179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:22.866808 26184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:22.868077 26184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:22.868474 26184 catalog_manager.cc:1259] Loaded cluster ID: 7e7d8ad83241492aa394b5f797ecb8bf
I20260928 02:42:22.868526 26184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:22.870040 26184 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:22.870244 26184 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Loaded TSK: 0
I20260928 02:42:22.870291 26184 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c3e371363084369b5ca68b537126c86: Loaded TSK: 1
F20260928 02:42:22.870553 26184 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8434e7b9ad google::LogMessage::Fail() at ??:0
@ 0x7f8434e7e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f8434e7b4de google::LogMessage::Flush() at ??:0
@ 0x7f8434e7ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f84375e8104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f84375e5d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f843cdda1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f843cdda48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f843cdda594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f843cda0c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f843cdaaad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8436d490d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8436d3e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8437ba26db start_thread at ??:0
@ 0x7f843397961f clone at ??:0
W20260928 02:42:22.967741 26176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:22.968075 26176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:22.968118 26176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:22.969447 26176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:22.969512 26176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:22.969568 26176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:42:22.971144 26176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:38935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=39625
--tserver_master_addrs=127.9.130.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:22.971414 26176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:22.971652 26176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:22.977854 26214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:22.977830 26215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:22.977830 26217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:22.978324 26176 server_base.cc:1034] running on GCE node
I20260928 02:42:22.978519 26176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:22.978806 26176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:22.979961 26176 hybrid_clock.cc:648] HybridClock initialized: now 1790563342979948 us; error 31 us; skew 500 ppm
Sep 28 02:42:22 dist-test-slave-kkrh krb5kdc[25034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:22.982726 26176 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:42:22.983160 26176 webserver.cc:460] Webserver started at http://127.9.130.1:39625/ using document root <none> and password file <none>
I20260928 02:42:22.983428 26176 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:22.983521 26176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:22.985054 26176 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:22.985802 26224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:22.985991 26176 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:22.986104 26176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "efa3ceee4e04453f9f76ef6a3e72faab"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "65aeeb12a763cce0adfc4758dfa712fe"
server_key_iv: "04ad9d8a5a6e61065a6f0ca43f22be6f"
server_key_version: "encryptionkey@0"
I20260928 02:42:22.986470 26176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:23.007375 26176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:23.009842 26176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:23.009989 26176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:23.010288 26176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:23.010766 26231 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:42:23.012791 26176 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:42:23.012861 26176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:42:23.012907 26176 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:42:23.013584 26176 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:42:23.013640 26176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:42:23.013718 26231 tablet_bootstrap.cc:492] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab: Bootstrap starting.
I20260928 02:42:23.015616 26231 log.cc:824] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:23.018554 26231 tablet_bootstrap.cc:492] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:42:23.018882 26231 tablet_bootstrap.cc:492] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab: Bootstrap complete.
I20260928 02:42:23.019037 26231 ts_tablet_manager.cc:1397] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260928 02:42:23.020778 26176 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:38935
I20260928 02:42:23.020884 26339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:38935 every 8 connection(s)
I20260928 02:42:23.021150 26176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:42:23.020913 26231 raft_consensus.cc:348] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:23.021225 26231 raft_consensus.cc:729] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa3ceee4e04453f9f76ef6a3e72faab, State: Initialized, Role: FOLLOWER
I20260928 02:42:23.021397 26231 consensus_queue.cc:260] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:23.021797 26231 ts_tablet_manager.cc:1428] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:42:23.024288 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 26176
I20260928 02:42:23.024559 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:33251
--local_ip_for_outbound_sockets=127.9.130.2
--tserver_master_addrs=127.9.130.62:43253
--webserver_port=36563
--webserver_interface=127.9.130.2
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:23.069969 26343 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:23.021818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:52053)
0928 02:42:23.021953 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:23.021991 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:23.022153 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:23.069762 (+ 47609us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
W20260928 02:42:23.070192 26340 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:43253 (0 consecutive failures): Network error: Failed to ping master at 127.9.130.62:43253: Client connection negotiation failed: client connection to 127.9.130.62:43253: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:42:23.070578 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:23.070455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:51539)
0928 02:42:23.070512 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:23.070557 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:42:23.070984 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:23.070895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:44983)
0928 02:42:23.070943 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:23.070968 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:42:23.071121 26340 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:42:23.108510 26344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:23.108763 26344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:23.108808 26344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:23.109972 26344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:23.110044 26344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:23.110090 26344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:42:23.111533 26344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:33251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=36563
--tserver_master_addrs=127.9.130.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:23.111802 26344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:23.112082 26344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:23.117157 26350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:23.117195 26349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:23.117408 26344 server_base.cc:1034] running on GCE node
W20260928 02:42:23.117172 26352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:23.117663 26344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:23.117911 26344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:23.119071 26344 hybrid_clock.cc:648] HybridClock initialized: now 1790563343119051 us; error 35 us; skew 500 ppm
Sep 28 02:42:23 dist-test-slave-kkrh krb5kdc[25034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563343, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:23.121457 26344 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:42:23.121933 26344 webserver.cc:460] Webserver started at http://127.9.130.2:36563/ using document root <none> and password file <none>
I20260928 02:42:23.122150 26344 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:23.122217 26344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:23.123626 26344 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:23.124369 26359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:23.124558 26344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:42:23.124673 26344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "0aac57c631024653aebf40e84862bff9"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "b1b66697727be14611a0187288422d76"
server_key_iv: "2d9bfb7f47f47dbc01620148f0bf7001"
server_key_version: "encryptionkey@0"
I20260928 02:42:23.125043 26344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:23.153641 26344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:23.155982 26344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:23.156184 26344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:23.156442 26344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:23.156762 26344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:23.156817 26344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:23.156859 26344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:23.156886 26344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:23.163014 26344 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:33251
I20260928 02:42:23.163059 26472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:33251 every 8 connection(s)
I20260928 02:42:23.163409 26344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:42:23.164470 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:23.164071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:41423)
0928 02:42:23.164247 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:23.164321 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
W20260928 02:42:23.164897 26473 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:43253 (0 consecutive failures): Network error: Failed to ping master at 127.9.130.62:43253: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
I20260928 02:42:23.165203 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:23.165075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:37173)
0928 02:42:23.165145 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:23.165180 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:42:23.165629 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:23.165529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:35335)
0928 02:42:23.165585 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:23.165611 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 02:42:23.165778 26473 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:42:23.169773 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 26344
I20260928 02:42:23.170065 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:44485
--local_ip_for_outbound_sockets=127.9.130.3
--tserver_master_addrs=127.9.130.62:43253
--webserver_port=40917
--webserver_interface=127.9.130.3
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:23.251976 26476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:23.252223 26476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:23.252267 26476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:23.253548 26476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:23.253620 26476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:23.253667 26476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:42:23.255247 26476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:44485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=40917
--tserver_master_addrs=127.9.130.62:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:23.255501 26476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:23.255725 26476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:23.260699 26484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:23.260686 26482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:23.260792 26476 server_base.cc:1034] running on GCE node
W20260928 02:42:23.260681 26481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:23.261132 26476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:23.261338 26476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:23.262478 26476 hybrid_clock.cc:648] HybridClock initialized: now 1790563343262452 us; error 44 us; skew 500 ppm
Sep 28 02:42:23 dist-test-slave-kkrh krb5kdc[25034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563343, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:23.265005 26476 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:42:23.265497 26476 webserver.cc:460] Webserver started at http://127.9.130.3:40917/ using document root <none> and password file <none>
I20260928 02:42:23.265682 26476 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:23.265722 26476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:23.267009 26476 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:23.267601 26491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:23.267805 26476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:23.267910 26476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "b826c1382ed445b9b9cc717e9ecee409"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-kkrh"
server_key: "b9c7711022d251f8c6d1ca7058d62dc4"
server_key_iv: "9d08afcb9e17a63fa71296e95936d1d2"
server_key_version: "encryptionkey@0"
I20260928 02:42:23.268291 26476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:23.277245 26476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:23.279088 26476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:23.279201 26476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:23.279367 26476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:23.279660 26476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:23.279693 26476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:23.279714 26476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:23.279760 26476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:23.285425 26476 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:44485
I20260928 02:42:23.285502 26604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:44485 every 8 connection(s)
I20260928 02:42:23.285806 26476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:42:23.286830 26607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:23.286479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:55853)
0928 02:42:23.286623 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:23.286691 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
W20260928 02:42:23.287231 26605 heartbeater.cc:644] Failed to heartbeat to 127.9.130.62:43253 (0 consecutive failures): Network error: Failed to ping master at 127.9.130.62:43253: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
I20260928 02:42:23.287524 26607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:23.287418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:43503)
0928 02:42:23.287464 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:23.287491 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 02:42:23.288148 26607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:23.288055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:58799)
0928 02:42:23.288108 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:23.288132 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 02:42:23.288314 26605 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:42:23.295400 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 26476
I20260928 02:42:24.071674 26609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:24.071462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:49791)
0928 02:42:24.071580 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:24.071616 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260928 02:42:24.166278 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:24.166090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:40415)
0928 02:42:24.166197 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:24.166226 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260928 02:42:24.288795 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:24.288585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:41629)
0928 02:42:24.288716 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:24.288751 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":54,"threads_started":1}
I20260928 02:42:24.694736 26612 raft_consensus.cc:482] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:42:24.694859 26612 raft_consensus.cc:504] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:24.695246 26612 leader_election.cc:288] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485), 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251)
Sep 28 02:42:24 dist-test-slave-kkrh krb5kdc[25034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.3@KRBTEST.COM
Sep 28 02:42:24 dist-test-slave-kkrh krb5kdc[25034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.2@KRBTEST.COM
I20260928 02:42:24.705771 26615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:24.695535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:57113 (local address 127.9.130.3:44485)
0928 02:42:24.695744 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:24.695765 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:24.695799 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:42:24.696237 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:24.696239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:24.696281 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:24.696426 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:24.697216 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:24.698077 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:24.699067 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:24.699464 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:24.699592 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:24.702102 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:24.702105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:24.702106 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:24.702115 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:24.704086 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:24.704577 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:24.704581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:24.704581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:24.704679 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:24.704948 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:24.704949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:24.704949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:24.705161 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:24.705255 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:24.705557 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 02:42:24.705674 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260928 02:42:24.705770 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:24.695559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:33251 (local address 127.9.130.1:39199)
0928 02:42:24.695636 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:24.695645 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:24.695687 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:24.696573 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:24.696574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:24.696842 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:24.697247 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:24.697249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:24.698353 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:24.698354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:24.698965 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:24.698967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:24.699591 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:24.699592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:24.699693 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:24.700341 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:24.700349 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:24.702361 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:24.704280 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:24.704281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:24.704281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:24.704525 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:24.704804 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:24.704805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:24.704805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:24.704868 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:24.705237 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:24.705243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:24.705373 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 02:42:24.705458 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:42:24.705652 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":14,"thread_start_us":36,"threads_started":1}
I20260928 02:42:24.705798 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:24.695701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:39199 (local address 127.9.130.2:33251)
0928 02:42:24.695877 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:24.695894 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:24.695917 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:42:24.696340 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:24.696342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:24.696380 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:24.696595 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:24.697380 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:24.698245 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:24.699082 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:24.699489 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:24.699620 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:24.702468 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:24.702479 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:24.702479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:24.702488 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:24.704196 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:24.704629 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:24.704631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:24.704632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:24.704732 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:24.704934 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:24.704934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:24.704935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:24.705162 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:24.705255 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:24.705537 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:42:24.705689 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260928 02:42:24.705860 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:24.695397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:44485 (local address 127.9.130.1:57113)
0928 02:42:24.695505 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:24.695516 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:24.695573 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:24.696479 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:24.696482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:24.696839 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:24.697081 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:24.697087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:24.698204 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:24.698206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:24.698964 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:24.698966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:24.699567 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:24.699568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:24.699691 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:24.700384 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:24.700389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:24.701969 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:24.704166 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:24.704173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:24.704176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:24.704501 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:24.704745 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:24.704746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:24.704746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:24.704866 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:24.705235 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:24.705242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:24.705373 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 02:42:24.705488 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:42:24.705732 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":131,"thread_start_us":52,"threads_started":1}
W20260928 02:42:24.707196 26226 leader_election.cc:341] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485): Not found: Tablet not found: 3296f1d9c4be4a6ca45e7765bc298b54
W20260928 02:42:24.707263 26226 leader_election.cc:341] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251): Not found: Tablet not found: 3296f1d9c4be4a6ca45e7765bc298b54
I20260928 02:42:24.707307 26226 leader_election.cc:302] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efa3ceee4e04453f9f76ef6a3e72faab; no voters: 0aac57c631024653aebf40e84862bff9, b826c1382ed445b9b9cc717e9ecee409
I20260928 02:42:24.707407 26612 raft_consensus.cc:2726] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:42:25.072177 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:25.072043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:54921)
0928 02:42:25.072106 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:25.072138 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 02:42:25.166915 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:25.166689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:39373)
0928 02:42:25.166819 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:25.166863 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260928 02:42:25.289397 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:25.289205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:51073)
0928 02:42:25.289328 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:25.289361 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
I20260928 02:42:26.072804 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:26.072601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:34613)
0928 02:42:26.072722 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:26.072758 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260928 02:42:26.167572 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:26.167383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:49419)
0928 02:42:26.167498 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:26.167531 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260928 02:42:26.290094 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:26.289827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:36109)
0928 02:42:26.289965 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:26.290053 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260928 02:42:26.378437 26622 raft_consensus.cc:482] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:42:26.378528 26622 raft_consensus.cc:504] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:26.378674 26622 leader_election.cc:288] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485), 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251)
W20260928 02:42:26.379067 26226 leader_election.cc:341] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485): Not found: Tablet not found: 3296f1d9c4be4a6ca45e7765bc298b54
W20260928 02:42:26.379142 26226 leader_election.cc:341] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251): Not found: Tablet not found: 3296f1d9c4be4a6ca45e7765bc298b54
I20260928 02:42:26.379159 26226 leader_election.cc:302] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efa3ceee4e04453f9f76ef6a3e72faab; no voters: 0aac57c631024653aebf40e84862bff9, b826c1382ed445b9b9cc717e9ecee409
I20260928 02:42:26.379226 26622 raft_consensus.cc:2726] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:42:27.073483 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:27.073280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:44799)
0928 02:42:27.073400 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:27.073438 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260928 02:42:27.168354 26624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:27.168154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:41475)
0928 02:42:27.168266 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:27.168311 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260928 02:42:27.290760 26625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:27.290568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:50223)
0928 02:42:27.290689 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:27.290724 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260928 02:42:27.987671 26626 raft_consensus.cc:482] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:42:27.987783 26626 raft_consensus.cc:504] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:27.988003 26626 leader_election.cc:288] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485), 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251)
W20260928 02:42:27.988384 26226 leader_election.cc:341] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251): Not found: Tablet not found: 3296f1d9c4be4a6ca45e7765bc298b54
W20260928 02:42:27.988461 26226 leader_election.cc:341] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485): Not found: Tablet not found: 3296f1d9c4be4a6ca45e7765bc298b54
I20260928 02:42:27.988476 26226 leader_election.cc:302] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efa3ceee4e04453f9f76ef6a3e72faab; no voters: 0aac57c631024653aebf40e84862bff9, b826c1382ed445b9b9cc717e9ecee409
I20260928 02:42:27.988581 26626 raft_consensus.cc:2726] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:42:28.074246 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:28.073986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:50359)
0928 02:42:28.074146 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:28.074197 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":57,"threads_started":1}
I20260928 02:42:28.169027 26628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:28.168827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:35887)
0928 02:42:28.168948 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:28.168985 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260928 02:42:28.291448 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:28.291251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:48113)
0928 02:42:28.291373 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:28.291407 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260928 02:42:29.074936 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:29.074713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:58919)
0928 02:42:29.074853 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:29.074891 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":57,"threads_started":1}
I20260928 02:42:29.169727 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:29.169491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:44167)
0928 02:42:29.169647 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:29.169685 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":118,"threads_started":1}
I20260928 02:42:29.292186 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:29.291985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:41041)
0928 02:42:29.292105 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:29.292146 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260928 02:42:30.075773 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:30.075535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:45281)
0928 02:42:30.075679 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:30.075726 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260928 02:42:30.091842 26634 raft_consensus.cc:482] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:42:30.091939 26634 raft_consensus.cc:504] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:30.092109 26634 leader_election.cc:288] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485), 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251)
W20260928 02:42:30.092603 26226 leader_election.cc:341] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485): Not found: Tablet not found: 3296f1d9c4be4a6ca45e7765bc298b54
W20260928 02:42:30.092689 26226 leader_election.cc:341] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251): Not found: Tablet not found: 3296f1d9c4be4a6ca45e7765bc298b54
I20260928 02:42:30.092707 26226 leader_election.cc:302] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efa3ceee4e04453f9f76ef6a3e72faab; no voters: 0aac57c631024653aebf40e84862bff9, b826c1382ed445b9b9cc717e9ecee409
I20260928 02:42:30.092789 26634 raft_consensus.cc:2726] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:42:30.170588 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:30.170251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:34041)
0928 02:42:30.170479 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:30.170539 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
I20260928 02:42:30.292871 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:30.292623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:32941)
0928 02:42:30.292762 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:30.292801 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":55,"threads_started":1}
I20260928 02:42:31.076520 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:31.076331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:35157)
0928 02:42:31.076444 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:31.076474 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260928 02:42:31.171407 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:31.171167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:48813)
0928 02:42:31.171325 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:31.171361 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260928 02:42:31.293614 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:31.293400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:58965)
0928 02:42:31.293536 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:31.293575 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260928 02:42:32.077117 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:32.076927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:58975)
0928 02:42:32.077038 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:32.077071 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260928 02:42:32.172191 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:32.171947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:36063)
0928 02:42:32.172066 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:32.172147 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260928 02:42:32.294286 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:32.294078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:37931)
0928 02:42:32.294205 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:32.294244 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260928 02:42:33.077744 26643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:33.077553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:48865)
0928 02:42:33.077666 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:33.077699 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260928 02:42:33.172801 26644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:33.172599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:40999)
0928 02:42:33.172719 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:33.172756 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260928 02:42:33.294975 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:33.294753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:59267)
0928 02:42:33.294887 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:33.294934 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260928 02:42:34.078413 26646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:34.078196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:47109)
0928 02:42:34.078331 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:34.078367 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":56,"threads_started":1}
I20260928 02:42:34.173450 26647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:34.173249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:57613)
0928 02:42:34.173374 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:34.173408 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260928 02:42:34.295632 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:34.295450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:49365)
0928 02:42:34.295559 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:34.295591 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260928 02:42:35.079100 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:35.078860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:50445)
0928 02:42:35.079023 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:35.079054 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260928 02:42:35.174172 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:35.173954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:35473)
0928 02:42:35.174090 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:35.174127 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260928 02:42:35.296284 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:35.296070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:50187)
0928 02:42:35.296209 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:35.296240 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":59,"threads_started":1}
I20260928 02:42:35.729424 26652 raft_consensus.cc:482] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:42:35.729545 26652 raft_consensus.cc:504] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b826c1382ed445b9b9cc717e9ecee409" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 44485 } } peers { permanent_uuid: "0aac57c631024653aebf40e84862bff9" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33251 } } peers { permanent_uuid: "efa3ceee4e04453f9f76ef6a3e72faab" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38935 } }
I20260928 02:42:35.729723 26652 leader_election.cc:288] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485), 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251)
W20260928 02:42:35.730141 26226 leader_election.cc:341] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b826c1382ed445b9b9cc717e9ecee409 (127.9.130.3:44485): Not found: Tablet not found: 3296f1d9c4be4a6ca45e7765bc298b54
W20260928 02:42:35.730206 26226 leader_election.cc:341] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0aac57c631024653aebf40e84862bff9 (127.9.130.2:33251): Not found: Tablet not found: 3296f1d9c4be4a6ca45e7765bc298b54
I20260928 02:42:35.730253 26226 leader_election.cc:302] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efa3ceee4e04453f9f76ef6a3e72faab; no voters: 0aac57c631024653aebf40e84862bff9, b826c1382ed445b9b9cc717e9ecee409
I20260928 02:42:35.730355 26652 raft_consensus.cc:2726] T 3296f1d9c4be4a6ca45e7765bc298b54 P efa3ceee4e04453f9f76ef6a3e72faab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:42:36.079793 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:36.079580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:59293)
0928 02:42:36.079710 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:36.079746 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260928 02:42:36.174875 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:36.174665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:48543)
0928 02:42:36.174795 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:36.174832 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260928 02:42:36.297111 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:36.296869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:49457)
0928 02:42:36.297015 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:36.297064 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260928 02:42:37.080468 26656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:37.080247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:58841)
0928 02:42:37.080373 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:37.080408 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260928 02:42:37.175500 26657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:37.175279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:58495)
0928 02:42:37.175405 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:37.175456 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260928 02:42:37.297904 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:37.297690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:38551)
0928 02:42:37.297826 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:37.297859 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260928 02:42:38.081136 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:38.080956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.1:39357)
0928 02:42:38.081063 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:38.081093 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260928 02:42:38.176208 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:38.176020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.2:40753)
0928 02:42:38.176130 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:38.176167 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260928 02:42:38.297550 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 26176
I20260928 02:42:38.298770 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:38.298422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:43253 (local address 127.9.130.3:60033)
0928 02:42:38.298692 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:38.298733 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.62:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":68,"threads_started":1}
I20260928 02:42:38.302275 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 26344
I20260928 02:42:38.306394 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 26476
2026-09-28T02:42:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58140 ms)
[----------] 30 tests from SecurityITest (231030 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:38 dist-test-slave-kkrh krb5kdc[26664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:38 dist-test-slave-kkrh krb5kdc[26664](info): set up 2 sockets
Sep 28 02:42:38 dist-test-slave-kkrh krb5kdc[26664](info): commencing operation
krb5kdc: starting...
W20260928 02:42:39.407506 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:42:39 dist-test-slave-kkrh krb5kdc[26664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:42:39.526808 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:40077
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:40077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:39.607008 26677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:39.607218 26677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:39.607244 26677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:39.608443 26677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:39.608484 26677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:39.608497 26677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:39.608510 26677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:39.608565 26677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:39.610008 26677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:40077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:40077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:39.610253 26677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:39.610481 26677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:42:39.616025 26677 server_base.cc:1034] running on GCE node
W20260928 02:42:39.616001 26685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:39.616001 26683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:39.616004 26682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:39.616494 26677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:39.616735 26677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:39.617887 26677 hybrid_clock.cc:648] HybridClock initialized: now 1790563359617871 us; error 33 us; skew 500 ppm
Sep 28 02:42:39 dist-test-slave-kkrh krb5kdc[26664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563359, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:39.620414 26677 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:42:39.620911 26677 webserver.cc:460] Webserver started at http://127.9.130.62:41303/ using document root <none> and password file <none>
I20260928 02:42:39.621150 26677 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:39.621217 26677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:39.621371 26677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:39.622460 26677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "45eeaa4e34a549f79237edbd4579b38d"
format_stamp: "Formatted at 2026-09-28 02:42:39 on dist-test-slave-kkrh"
server_key: "1f32d2c2a3e536a0e160436f65697ac7"
server_key_iv: "8f21233f6724943d6f84727b83c7f5f1"
server_key_version: "encryptionkey@0"
I20260928 02:42:39.622823 26677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "45eeaa4e34a549f79237edbd4579b38d"
format_stamp: "Formatted at 2026-09-28 02:42:39 on dist-test-slave-kkrh"
server_key: "1f32d2c2a3e536a0e160436f65697ac7"
server_key_iv: "8f21233f6724943d6f84727b83c7f5f1"
server_key_version: "encryptionkey@0"
I20260928 02:42:39.624186 26677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:42:39.624888 26692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:39.625051 26677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:42:39.625154 26677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "45eeaa4e34a549f79237edbd4579b38d"
format_stamp: "Formatted at 2026-09-28 02:42:39 on dist-test-slave-kkrh"
server_key: "1f32d2c2a3e536a0e160436f65697ac7"
server_key_iv: "8f21233f6724943d6f84727b83c7f5f1"
server_key_version: "encryptionkey@0"
I20260928 02:42:39.625224 26677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:39.653487 26677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:39.655969 26677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:39.656154 26677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:39.660351 26677 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:40077
I20260928 02:42:39.660382 26744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:40077 every 8 connection(s)
I20260928 02:42:39.660740 26677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:39.661342 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 26677
I20260928 02:42:39.661456 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:39.661644 9736 external_mini_cluster.cc:1428] Setting key 3518f8e889cf1c8acb4a69454f4350ed
I20260928 02:42:39.661789 26745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:39.665285 26745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d: Bootstrap starting.
I20260928 02:42:39.666007 26745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:39.666333 26745 log.cc:824] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:39.667110 26745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d: No bootstrap required, opened a new log
Sep 28 02:42:39 dist-test-slave-kkrh krb5kdc[26664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:39.669035 26745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45eeaa4e34a549f79237edbd4579b38d" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 40077 } }
I20260928 02:42:39.669198 26745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:39.669247 26745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45eeaa4e34a549f79237edbd4579b38d, State: Initialized, Role: FOLLOWER
I20260928 02:42:39.669410 26745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45eeaa4e34a549f79237edbd4579b38d" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 40077 } }
I20260928 02:42:39.669507 26745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:39.669550 26745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:39.669581 26745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:39.670289 26745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45eeaa4e34a549f79237edbd4579b38d" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 40077 } }
I20260928 02:42:39.670410 26745 leader_election.cc:302] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45eeaa4e34a549f79237edbd4579b38d; no voters:
I20260928 02:42:39.670586 26745 leader_election.cc:288] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:39.670660 26750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:39.670831 26745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:39.670837 26750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [term 1 LEADER]: Becoming Leader. State: Replica: 45eeaa4e34a549f79237edbd4579b38d, State: Running, Role: LEADER
I20260928 02:42:39.670957 26750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45eeaa4e34a549f79237edbd4579b38d" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 40077 } }
I20260928 02:42:39.671275 26751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45eeaa4e34a549f79237edbd4579b38d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45eeaa4e34a549f79237edbd4579b38d" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 40077 } } }
I20260928 02:42:39.671360 26751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:39.671335 26752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45eeaa4e34a549f79237edbd4579b38d. Latest consensus state: current_term: 1 leader_uuid: "45eeaa4e34a549f79237edbd4579b38d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45eeaa4e34a549f79237edbd4579b38d" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 40077 } } }
I20260928 02:42:39.671736 26752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:39.671998 26759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:39.672523 26759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:39.673602 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:39.662203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.9.130.62:40077)
0928 02:42:39.662387 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:39.662396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:39.662430 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:42:39.662958 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:39.662964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:39.663008 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:39.663122 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:39.664037 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:39.664929 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:39.666110 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:39.666597 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:39.666690 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:39.669229 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:39.669233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:39.669233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:39.669245 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:39.671235 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:39.671731 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:39.671733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:39.671734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:39.672069 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:39.672365 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:39.672365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:39.672366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:39.672688 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:39.673186 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:39.673273 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:42:39.673406 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
I20260928 02:42:39.674342 26759 catalog_manager.cc:1347] Generated new cluster ID: a6ccb5dcb8824726a8eac6f45deae21c
I20260928 02:42:39.674391 26759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:39.683853 26759 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:39.684341 26759 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:39.687320 26759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d: Generated new TSK 0
I20260928 02:42:39.687479 26759 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:42:39.718566 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:40077
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:39.801849 26771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:39.802054 26771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:39.802078 26771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:39.803303 26771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:39.803341 26771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:39.803366 26771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:42:39.804841 26771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:40077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:39.805095 26771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:39.805332 26771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:39.810261 26776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:39.810314 26779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:39.810289 26777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:39.810453 26771 server_base.cc:1034] running on GCE node
I20260928 02:42:39.810645 26771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:39.810838 26771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:39.812000 26771 hybrid_clock.cc:648] HybridClock initialized: now 1790563359811989 us; error 31 us; skew 500 ppm
Sep 28 02:42:39 dist-test-slave-kkrh krb5kdc[26664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563359, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:39.814368 26771 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:42:39.814826 26771 webserver.cc:460] Webserver started at http://127.9.130.1:36079/ using document root <none> and password file <none>
I20260928 02:42:39.815068 26771 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:39.815140 26771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:39.815289 26771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:39.816387 26771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "ee8c55492d09425d804312f435e025dc"
format_stamp: "Formatted at 2026-09-28 02:42:39 on dist-test-slave-kkrh"
server_key: "d46d9494faba36eb196ffbf963be23f5"
server_key_iv: "3f3f8fddc11c515f1b14c6a123c89788"
server_key_version: "encryptionkey@0"
I20260928 02:42:39.816747 26771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "ee8c55492d09425d804312f435e025dc"
format_stamp: "Formatted at 2026-09-28 02:42:39 on dist-test-slave-kkrh"
server_key: "d46d9494faba36eb196ffbf963be23f5"
server_key_iv: "3f3f8fddc11c515f1b14c6a123c89788"
server_key_version: "encryptionkey@0"
I20260928 02:42:39.818111 26771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:42:39.818954 26786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:39.819193 26771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:39.819300 26771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "ee8c55492d09425d804312f435e025dc"
format_stamp: "Formatted at 2026-09-28 02:42:39 on dist-test-slave-kkrh"
server_key: "d46d9494faba36eb196ffbf963be23f5"
server_key_iv: "3f3f8fddc11c515f1b14c6a123c89788"
server_key_version: "encryptionkey@0"
I20260928 02:42:39.819383 26771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:39.831111 26771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:39.833429 26771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:39.833617 26771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:39.833876 26771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:39.834213 26771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:39.834275 26771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:39.834321 26771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:39.834373 26771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:39.840664 26771 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:38949
I20260928 02:42:39.840701 26899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:38949 every 8 connection(s)
I20260928 02:42:39.841056 26771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:42:39.843477 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 26771
I20260928 02:42:39.843593 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:42:39.843871 9736 external_mini_cluster.cc:1428] Setting key fe47bebed0901cc13345d1d3499409df
Sep 28 02:42:39 dist-test-slave-kkrh krb5kdc[26664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563359, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:39.851701 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:39.841812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:41353 (local address 127.9.130.62:40077)
0928 02:42:39.841873 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:39.841875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:39.842064 (+ 189us) server_negotiation.cc:407] Connection header received
0928 02:42:39.842606 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:39.842607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:39.842618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:39.842681 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:39.843766 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:39.844565 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:39.845699 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:39.845945 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:39.846003 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:39.848708 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:39.848709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:39.848710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:39.848716 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:39.850202 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:39.850715 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:39.850716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:39.850716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:39.850768 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:39.851063 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:39.851064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:39.851064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:39.851166 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:39.851220 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:39.851561 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 02:42:39.851631 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:42:39.851986 26902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:39.841757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:40077 (local address 127.9.130.1:41353)
0928 02:42:39.841938 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:39.841985 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:39.842514 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:39.842768 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:39.842772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:39.843400 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:39.843649 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:39.843657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:39.844663 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:39.844664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:39.845581 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:39.845584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:39.846038 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:39.846039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:39.846203 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:39.846742 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:39.846752 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:39.848597 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:39.850311 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:39.850318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:39.850320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:39.850632 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:39.850839 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:39.850840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:39.850841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:39.850989 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:39.851235 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:39.851242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:39.851419 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 02:42:39.851540 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 02:42:39.851710 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
I20260928 02:42:39.852588 26900 heartbeater.cc:347] Connected to a master server at 127.9.130.62:40077
I20260928 02:42:39.852694 26900 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:39.852875 26900 heartbeater.cc:511] Master 127.9.130.62:40077 requested a full tablet report, sending...
I20260928 02:42:39.853408 26709 ts_manager.cc:194] Registered new tserver with Master: ee8c55492d09425d804312f435e025dc (127.9.130.1:38949)
I20260928 02:42:39.854103 26709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:41353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:42:39.871130 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:40077
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:39.952540 26905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:39.952728 26905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:39.952750 26905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:39.953819 26905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:39.953853 26905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:39.953879 26905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:42:39.955245 26905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:40077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:39.955492 26905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:39.955677 26905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:39.960595 26910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:39.960604 26913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:39.960688 26905 server_base.cc:1034] running on GCE node
W20260928 02:42:39.960616 26911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:39.960963 26905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:39.961187 26905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:39.962354 26905 hybrid_clock.cc:648] HybridClock initialized: now 1790563359962332 us; error 39 us; skew 500 ppm
Sep 28 02:42:39 dist-test-slave-kkrh krb5kdc[26664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563359, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:39.964844 26905 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:42:39.965282 26905 webserver.cc:460] Webserver started at http://127.9.130.2:33545/ using document root <none> and password file <none>
I20260928 02:42:39.965507 26905 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:39.965569 26905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:39.965657 26905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:39.966624 26905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "4166b56e8f45406f864f2a4f2e2bc2a1"
format_stamp: "Formatted at 2026-09-28 02:42:39 on dist-test-slave-kkrh"
server_key: "9deafcc3a3fd6af8395a15de216dc47b"
server_key_iv: "40e5653741d026388cfca28fa2d6ad84"
server_key_version: "encryptionkey@0"
I20260928 02:42:39.966912 26905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "4166b56e8f45406f864f2a4f2e2bc2a1"
format_stamp: "Formatted at 2026-09-28 02:42:39 on dist-test-slave-kkrh"
server_key: "9deafcc3a3fd6af8395a15de216dc47b"
server_key_iv: "40e5653741d026388cfca28fa2d6ad84"
server_key_version: "encryptionkey@0"
I20260928 02:42:39.968207 26905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:42:39.968961 26920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:39.969274 26905 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:42:39.969363 26905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "4166b56e8f45406f864f2a4f2e2bc2a1"
format_stamp: "Formatted at 2026-09-28 02:42:39 on dist-test-slave-kkrh"
server_key: "9deafcc3a3fd6af8395a15de216dc47b"
server_key_iv: "40e5653741d026388cfca28fa2d6ad84"
server_key_version: "encryptionkey@0"
I20260928 02:42:39.969419 26905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:39.989671 26905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:39.991523 26905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:39.991665 26905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:39.991879 26905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:39.992205 26905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:39.992239 26905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:39.992262 26905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:39.992314 26905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:39.997962 26905 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:46709
I20260928 02:42:39.998040 27033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:46709 every 8 connection(s)
I20260928 02:42:39.998368 26905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:42:40 dist-test-slave-kkrh krb5kdc[26664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563359, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:40.006138 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 26905
I20260928 02:42:40.006253 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:42:40.006462 9736 external_mini_cluster.cc:1428] Setting key b7c0d6e989d740d213703ff40b47ee51
I20260928 02:42:40.008378 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:39.999101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:43915 (local address 127.9.130.62:40077)
0928 02:42:39.999181 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:39.999183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:39.999376 (+ 193us) server_negotiation.cc:407] Connection header received
0928 02:42:39.999941 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:39.999943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:39.999952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:39.999995 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:40.000953 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.001462 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.002429 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.002660 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.002721 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.004987 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:40.004989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:40.004990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:40.004998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:40.006472 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:40.007236 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:40.007237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:40.007237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:40.007322 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:40.007585 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:40.007586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:40.007586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:40.007728 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:40.007805 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.008139 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.008237 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:42:40.008387 27036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:39.999052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:40077 (local address 127.9.130.2:43915)
0928 02:42:39.999225 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:39.999261 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:39.999783 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:40.000059 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:40.000063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:40.000623 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:40.000838 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.000842 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.001542 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.001543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.002337 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.002339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.002758 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.002759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.002867 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.003418 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:40.003427 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:40.004852 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:40.006777 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:40.006784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:40.006786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:40.007159 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:40.007401 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:40.007402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:40.007403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:40.007515 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:40.007801 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:40.007806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:40.007986 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 02:42:40.008094 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:42:40.008224 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260928 02:42:40.008970 27034 heartbeater.cc:347] Connected to a master server at 127.9.130.62:40077
I20260928 02:42:40.009078 27034 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:40.009275 27034 heartbeater.cc:511] Master 127.9.130.62:40077 requested a full tablet report, sending...
I20260928 02:42:40.009681 26709 ts_manager.cc:194] Registered new tserver with Master: 4166b56e8f45406f864f2a4f2e2bc2a1 (127.9.130.2:46709)
I20260928 02:42:40.010223 26709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:43915
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:42:40.034663 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:40077
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:40.114171 27039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:40.114399 27039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:40.114436 27039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:40.115550 27039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:40.115617 27039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:40.115661 27039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:42:40.117082 27039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:40077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:40.117347 27039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:40.117569 27039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:40.122505 27047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:40.122524 27044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:40.122723 27039 server_base.cc:1034] running on GCE node
W20260928 02:42:40.122567 27045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:40.122951 27039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:40.123186 27039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:40.124341 27039 hybrid_clock.cc:648] HybridClock initialized: now 1790563360124326 us; error 27 us; skew 500 ppm
Sep 28 02:42:40 dist-test-slave-kkrh krb5kdc[26664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563360, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:40.126732 27039 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:42:40.127204 27039 webserver.cc:460] Webserver started at http://127.9.130.3:41891/ using document root <none> and password file <none>
I20260928 02:42:40.127430 27039 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:40.127494 27039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:40.127650 27039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:40.128734 27039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "b50b6587b4654cd693ba1a49a650aa04"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "2709b8f34ee4d074895d5d493fc6c39d"
server_key_iv: "eec752e45fe09076820af191ee437738"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.129129 27039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "b50b6587b4654cd693ba1a49a650aa04"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "2709b8f34ee4d074895d5d493fc6c39d"
server_key_iv: "eec752e45fe09076820af191ee437738"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.130396 27039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:42:40.131176 27054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:40.131372 27039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:40.131490 27039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "b50b6587b4654cd693ba1a49a650aa04"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "2709b8f34ee4d074895d5d493fc6c39d"
server_key_iv: "eec752e45fe09076820af191ee437738"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.131585 27039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:40.141431 27039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:40.143378 27039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:40.143572 27039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:40.143800 27039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:40.144212 27039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:40.144269 27039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:40.144312 27039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:40.144344 27039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:40.150950 27039 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:36689
I20260928 02:42:40.150985 27167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:36689 every 8 connection(s)
I20260928 02:42:40.151312 27039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:42:40 dist-test-slave-kkrh krb5kdc[26664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563360, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:40.159521 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 27039
I20260928 02:42:40.159644 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:42:40.159873 9736 external_mini_cluster.cc:1428] Setting key 0d2392d964cefa5ea377776315ece9b7
I20260928 02:42:40.160565 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.152076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:41507 (local address 127.9.130.62:40077)
0928 02:42:40.152122 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.152124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.152353 (+ 229us) server_negotiation.cc:407] Connection header received
0928 02:42:40.152891 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.152893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.152905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.152969 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:40.153910 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.154424 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.155432 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.155641 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.155729 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.157875 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:40.157877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:40.157877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:40.157885 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:40.159058 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:40.159468 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:40.159469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:40.159470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:40.159529 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:40.159838 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:40.159839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:40.159839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:40.159987 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:40.160062 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.160393 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.160477 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:42:40.160761 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.152057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:40077 (local address 127.9.130.3:41507)
0928 02:42:40.152236 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:40.152270 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:40.152784 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:40.153037 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:40.153042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:40.153599 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:40.153813 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.153818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.154521 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.154523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.155335 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.155337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.155740 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.155742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.155888 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.156385 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:40.156392 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:40.157780 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:40.159157 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:40.159164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:40.159166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:40.159381 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:40.159602 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:40.159603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:40.159603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:40.159756 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:40.160068 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:40.160074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:40.160232 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 02:42:40.160366 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 02:42:40.160520 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260928 02:42:40.161206 27168 heartbeater.cc:347] Connected to a master server at 127.9.130.62:40077
I20260928 02:42:40.161291 27168 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:40.161427 27168 heartbeater.cc:511] Master 127.9.130.62:40077 requested a full tablet report, sending...
I20260928 02:42:40.161780 26709 ts_manager.cc:194] Registered new tserver with Master: b50b6587b4654cd693ba1a49a650aa04 (127.9.130.3:36689)
I20260928 02:42:40.162150 26709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:41507
I20260928 02:42:40.172317 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:42:40.179031 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.173772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.9.130.62:40077)
0928 02:42:40.173823 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.173825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.173860 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:42:40.173925 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.173926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.173934 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.173977 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:40.174462 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.174934 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.175631 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.175855 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.175919 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.176964 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:40.176965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:40.176966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:40.176971 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:40.178119 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:40.178411 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:40.178412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:40.178412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:40.178460 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:40.178618 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:40.178618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:40.178618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:40.178758 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:40.178836 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.178913 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.178971 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:42:40.180526 26709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:42:40.182075 26709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:42:40.188712 27177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.184536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:38949 (local address 127.0.0.1:53980)
0928 02:42:40.184760 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:40.184769 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:40.184818 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:40.185120 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:40.185123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:40.185125 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:40.185195 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.185197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.186168 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.186169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.187393 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.187396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.188417 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.188419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.188491 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.188507 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 02:42:40.188579 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:42:40.188632 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":44,"threads_started":1}
I20260928 02:42:40.188743 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.184676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.9.130.1:38949)
0928 02:42:40.184860 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.184864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.184876 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:42:40.184920 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.184921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.184937 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.185023 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:40.185290 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.186040 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.187478 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.188326 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.188405 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.188464 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.188616 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.188659 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":72,"threads_started":1}
I20260928 02:42:40.189107 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.184342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:46709 (local address 127.0.0.1:56296)
0928 02:42:40.184691 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:40.184739 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:40.184818 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:40.185231 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:40.185232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:40.185235 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:40.185272 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.185274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.186526 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.186527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.187577 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.187579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.188840 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.188841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.188926 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.188929 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:40.188997 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:42:40.189036 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"spinlock_wait_cycles":896,"thread_start_us":60,"threads_started":1}
I20260928 02:42:40.189363 27178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.184468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.9.130.2:46709)
0928 02:42:40.185007 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.185012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.185022 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:42:40.185061 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.185063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.185078 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.185153 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:40.185344 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.186411 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.187656 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.188335 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.188405 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.188464 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.189219 (+ 755us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.189271 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":114,"threads_started":1}
I20260928 02:42:40.189541 26834 tablet_service.cc:1467] Processing CreateTablet for tablet 691cd312ab2e433c8cc7f13df0fa5698 (DEFAULT_TABLE table=test-table [id=c72cc332df6e48009c4b9725f42894c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:40.189793 27176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.184341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:36689 (local address 127.0.0.1:53762)
0928 02:42:40.184718 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:40.184759 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:40.184818 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:40.185135 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:40.185136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:40.185138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:40.185209 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.185211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.187656 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.187656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.188491 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.188492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.189558 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.189559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.189620 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.189623 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:40.189676 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 02:42:40.189728 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260928 02:42:40.189849 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.184513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.9.130.3:36689)
0928 02:42:40.184666 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.184671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.184898 (+ 227us) server_negotiation.cc:407] Connection header received
0928 02:42:40.184939 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.184941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.184963 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.185053 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:40.185292 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.186411 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.188599 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.189406 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.189470 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.189522 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.189715 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.189762 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260928 02:42:40.189908 26834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691cd312ab2e433c8cc7f13df0fa5698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:40.189999 26968 tablet_service.cc:1467] Processing CreateTablet for tablet 691cd312ab2e433c8cc7f13df0fa5698 (DEFAULT_TABLE table=test-table [id=c72cc332df6e48009c4b9725f42894c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:40.190249 26968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691cd312ab2e433c8cc7f13df0fa5698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:40.192440 27182 tablet_bootstrap.cc:492] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc: Bootstrap starting.
I20260928 02:42:40.192704 27183 tablet_bootstrap.cc:492] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1: Bootstrap starting.
I20260928 02:42:40.193130 27182 tablet_bootstrap.cc:654] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:40.193382 27183 tablet_bootstrap.cc:654] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:40.193393 27102 tablet_service.cc:1467] Processing CreateTablet for tablet 691cd312ab2e433c8cc7f13df0fa5698 (DEFAULT_TABLE table=test-table [id=c72cc332df6e48009c4b9725f42894c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:40.193729 27102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691cd312ab2e433c8cc7f13df0fa5698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:40.193719 27182 log.cc:824] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:40.194041 27183 log.cc:824] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:40.194505 27182 tablet_bootstrap.cc:492] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc: No bootstrap required, opened a new log
I20260928 02:42:40.194579 27182 ts_tablet_manager.cc:1397] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:40.194780 27183 tablet_bootstrap.cc:492] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1: No bootstrap required, opened a new log
I20260928 02:42:40.194864 27183 ts_tablet_manager.cc:1397] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:40.196434 27183 raft_consensus.cc:348] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4166b56e8f45406f864f2a4f2e2bc2a1" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46709 } } peers { permanent_uuid: "b50b6587b4654cd693ba1a49a650aa04" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36689 } } peers { permanent_uuid: "ee8c55492d09425d804312f435e025dc" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38949 } }
I20260928 02:42:40.196615 27183 raft_consensus.cc:374] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:40.196663 27183 raft_consensus.cc:729] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4166b56e8f45406f864f2a4f2e2bc2a1, State: Initialized, Role: FOLLOWER
I20260928 02:42:40.196647 27182 raft_consensus.cc:348] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4166b56e8f45406f864f2a4f2e2bc2a1" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46709 } } peers { permanent_uuid: "b50b6587b4654cd693ba1a49a650aa04" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36689 } } peers { permanent_uuid: "ee8c55492d09425d804312f435e025dc" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38949 } }
I20260928 02:42:40.196779 27183 consensus_queue.cc:260] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4166b56e8f45406f864f2a4f2e2bc2a1" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46709 } } peers { permanent_uuid: "b50b6587b4654cd693ba1a49a650aa04" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36689 } } peers { permanent_uuid: "ee8c55492d09425d804312f435e025dc" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38949 } }
I20260928 02:42:40.196861 27182 raft_consensus.cc:374] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:40.196907 27182 raft_consensus.cc:729] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8c55492d09425d804312f435e025dc, State: Initialized, Role: FOLLOWER
I20260928 02:42:40.197031 27182 consensus_queue.cc:260] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4166b56e8f45406f864f2a4f2e2bc2a1" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46709 } } peers { permanent_uuid: "b50b6587b4654cd693ba1a49a650aa04" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36689 } } peers { permanent_uuid: "ee8c55492d09425d804312f435e025dc" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38949 } }
I20260928 02:42:40.197096 27183 ts_tablet_manager.cc:1428] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:40.197185 27186 tablet_bootstrap.cc:492] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04: Bootstrap starting.
I20260928 02:42:40.197233 27034 heartbeater.cc:503] Master 127.9.130.62:40077 was elected leader, sending a full tablet report...
I20260928 02:42:40.197651 27182 ts_tablet_manager.cc:1428] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:42:40.197651 26900 heartbeater.cc:503] Master 127.9.130.62:40077 was elected leader, sending a full tablet report...
I20260928 02:42:40.197898 27186 tablet_bootstrap.cc:654] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:40.198518 27186 log.cc:824] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:40.199188 27186 tablet_bootstrap.cc:492] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04: No bootstrap required, opened a new log
I20260928 02:42:40.199275 27186 ts_tablet_manager.cc:1397] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:40.200892 27186 raft_consensus.cc:348] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4166b56e8f45406f864f2a4f2e2bc2a1" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46709 } } peers { permanent_uuid: "b50b6587b4654cd693ba1a49a650aa04" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36689 } } peers { permanent_uuid: "ee8c55492d09425d804312f435e025dc" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38949 } }
I20260928 02:42:40.201053 27186 raft_consensus.cc:374] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:40.201102 27186 raft_consensus.cc:729] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50b6587b4654cd693ba1a49a650aa04, State: Initialized, Role: FOLLOWER
I20260928 02:42:40.201220 27186 consensus_queue.cc:260] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4166b56e8f45406f864f2a4f2e2bc2a1" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46709 } } peers { permanent_uuid: "b50b6587b4654cd693ba1a49a650aa04" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36689 } } peers { permanent_uuid: "ee8c55492d09425d804312f435e025dc" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38949 } }
I20260928 02:42:40.201504 27186 ts_tablet_manager.cc:1428] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:40.201524 27168 heartbeater.cc:503] Master 127.9.130.62:40077 was elected leader, sending a full tablet report...
W20260928 02:42:40.248988 27035 tablet.cc:2369] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:42:40.341866 26901 tablet.cc:2369] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:42:40.357519 27188 raft_consensus.cc:482] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:42:40.357649 27188 raft_consensus.cc:504] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4166b56e8f45406f864f2a4f2e2bc2a1" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46709 } } peers { permanent_uuid: "b50b6587b4654cd693ba1a49a650aa04" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36689 } } peers { permanent_uuid: "ee8c55492d09425d804312f435e025dc" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38949 } }
I20260928 02:42:40.358062 27188 leader_election.cc:288] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4166b56e8f45406f864f2a4f2e2bc2a1 (127.9.130.2:46709), b50b6587b4654cd693ba1a49a650aa04 (127.9.130.3:36689)
I20260928 02:42:40.361259 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.358388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:37523 (local address 127.9.130.3:36689)
0928 02:42:40.358437 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.358440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.358553 (+ 113us) server_negotiation.cc:407] Connection header received
0928 02:42:40.358644 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.358645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.358656 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.358737 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:40.358951 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.359402 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.360475 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.360895 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.360963 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.360974 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.361157 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.361193 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:42:40.361260 27178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.358388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:38847 (local address 127.9.130.2:46709)
0928 02:42:40.358445 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.358448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.358553 (+ 105us) server_negotiation.cc:407] Connection header received
0928 02:42:40.358636 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.358638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.358651 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.358735 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:40.358906 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.359388 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.360476 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.360916 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.360984 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.360993 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.361153 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.361188 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:42:40.361285 27191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.358296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:46709 (local address 127.9.130.1:38847)
0928 02:42:40.358455 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:40.358467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:40.358566 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:40.358740 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:40.358741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:40.358743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:40.358835 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.358837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.359485 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.359487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.360385 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.360388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.360986 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.360987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.361060 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.361063 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:40.361146 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:42:40.361188 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":38,"thread_start_us":81,"threads_started":1}
I20260928 02:42:40.361294 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.358296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:36689 (local address 127.9.130.1:37523)
0928 02:42:40.358466 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:40.358476 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:40.358568 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:40.358743 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:40.358744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:40.358746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:40.358857 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.358859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.359508 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.359509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.360385 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.360387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.360965 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.360966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.361065 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.361069 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:42:40.361149 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:42:40.361190 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":23,"spinlock_wait_cycles":12672,"thread_start_us":82,"threads_started":1}
I20260928 02:42:40.361635 26988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691cd312ab2e433c8cc7f13df0fa5698" candidate_uuid: "ee8c55492d09425d804312f435e025dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4166b56e8f45406f864f2a4f2e2bc2a1" is_pre_election: true
I20260928 02:42:40.361791 26988 raft_consensus.cc:2454] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee8c55492d09425d804312f435e025dc in term 0.
I20260928 02:42:40.362051 26788 leader_election.cc:302] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4166b56e8f45406f864f2a4f2e2bc2a1, ee8c55492d09425d804312f435e025dc; no voters:
I20260928 02:42:40.362088 27122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691cd312ab2e433c8cc7f13df0fa5698" candidate_uuid: "ee8c55492d09425d804312f435e025dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b50b6587b4654cd693ba1a49a650aa04" is_pre_election: true
I20260928 02:42:40.362222 27122 raft_consensus.cc:2454] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee8c55492d09425d804312f435e025dc in term 0.
I20260928 02:42:40.362214 27188 raft_consensus.cc:2781] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:42:40.362294 27188 raft_consensus.cc:482] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:42:40.362329 27188 raft_consensus.cc:3037] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:40.363044 27188 raft_consensus.cc:504] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4166b56e8f45406f864f2a4f2e2bc2a1" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46709 } } peers { permanent_uuid: "b50b6587b4654cd693ba1a49a650aa04" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36689 } } peers { permanent_uuid: "ee8c55492d09425d804312f435e025dc" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38949 } }
I20260928 02:42:40.363191 27188 leader_election.cc:288] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [CANDIDATE]: Term 1 election: Requested vote from peers 4166b56e8f45406f864f2a4f2e2bc2a1 (127.9.130.2:46709), b50b6587b4654cd693ba1a49a650aa04 (127.9.130.3:36689)
I20260928 02:42:40.363461 26988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691cd312ab2e433c8cc7f13df0fa5698" candidate_uuid: "ee8c55492d09425d804312f435e025dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4166b56e8f45406f864f2a4f2e2bc2a1"
I20260928 02:42:40.363521 27122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691cd312ab2e433c8cc7f13df0fa5698" candidate_uuid: "ee8c55492d09425d804312f435e025dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b50b6587b4654cd693ba1a49a650aa04"
I20260928 02:42:40.363570 26988 raft_consensus.cc:3037] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:40.363583 27122 raft_consensus.cc:3037] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:40.364471 26988 raft_consensus.cc:2454] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8c55492d09425d804312f435e025dc in term 1.
I20260928 02:42:40.364477 27122 raft_consensus.cc:2454] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8c55492d09425d804312f435e025dc in term 1.
I20260928 02:42:40.364665 26788 leader_election.cc:302] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4166b56e8f45406f864f2a4f2e2bc2a1, ee8c55492d09425d804312f435e025dc; no voters:
I20260928 02:42:40.364789 27188 raft_consensus.cc:2781] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:40.364943 27188 raft_consensus.cc:686] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [term 1 LEADER]: Becoming Leader. State: Replica: ee8c55492d09425d804312f435e025dc, State: Running, Role: LEADER
I20260928 02:42:40.365044 27188 consensus_queue.cc:237] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4166b56e8f45406f864f2a4f2e2bc2a1" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46709 } } peers { permanent_uuid: "b50b6587b4654cd693ba1a49a650aa04" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36689 } } peers { permanent_uuid: "ee8c55492d09425d804312f435e025dc" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38949 } }
I20260928 02:42:40.365885 26709 catalog_manager.cc:5495] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc reported cstate change: term changed from 0 to 1, leader changed from <none> to ee8c55492d09425d804312f435e025dc (127.9.130.1). New cstate: current_term: 1 leader_uuid: "ee8c55492d09425d804312f435e025dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4166b56e8f45406f864f2a4f2e2bc2a1" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b50b6587b4654cd693ba1a49a650aa04" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee8c55492d09425d804312f435e025dc" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38949 } health_report { overall_health: HEALTHY } } }
I20260928 02:42:40.376025 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.373885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53988 (local address 127.9.130.1:38949)
0928 02:42:40.373945 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.373947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.373953 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:42:40.373978 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.373979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.373995 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.374054 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:42:40.374182 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.374617 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.375336 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.375507 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.375556 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.375683 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:42:40.375768 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:42:40.375878 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.375924 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.375940 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:42:40.377614 26988 raft_consensus.cc:1261] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1 [term 1 FOLLOWER]: Refusing update from remote peer ee8c55492d09425d804312f435e025dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:40.377709 27122 raft_consensus.cc:1261] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04 [term 1 FOLLOWER]: Refusing update from remote peer ee8c55492d09425d804312f435e025dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:40.377974 27188 consensus_queue.cc:1035] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4166b56e8f45406f864f2a4f2e2bc2a1" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 46709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:40.378088 27193 consensus_queue.cc:1035] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b50b6587b4654cd693ba1a49a650aa04" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:40.380774 27197 mvcc.cc:205] Tried to move back new op lower bound from 7334147524104331264 to 7334147524055953408. Current Snapshot: MvccSnapshot[applied={T|T < 7334147524104331264}]
I20260928 02:42:40.381093 27198 mvcc.cc:205] Tried to move back new op lower bound from 7334147524104331264 to 7334147524055953408. Current Snapshot: MvccSnapshot[applied={T|T < 7334147524104331264}]
I20260928 02:42:40.383138 26709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:40.383229 26709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:40.383981 26709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45eeaa4e34a549f79237edbd4579b38d: Sending DeleteTablet for 3 replicas of tablet 691cd312ab2e433c8cc7f13df0fa5698
I20260928 02:42:40.384331 26968 tablet_service.cc:1514] Processing DeleteTablet for tablet 691cd312ab2e433c8cc7f13df0fa5698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:40 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:56296
I20260928 02:42:40.384476 26834 tablet_service.cc:1514] Processing DeleteTablet for tablet 691cd312ab2e433c8cc7f13df0fa5698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:40 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:53980
I20260928 02:42:40.384629 27206 tablet_replica.cc:330] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1: stopping tablet replica
I20260928 02:42:40.384634 27102 tablet_service.cc:1514] Processing DeleteTablet for tablet 691cd312ab2e433c8cc7f13df0fa5698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:40 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:53762
I20260928 02:42:40.384793 27206 raft_consensus.cc:2229] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:42:40.384841 27207 tablet_replica.cc:330] T 691cd312ab2e433c8cc7f13df0fa5698 P ee8c55492d09425d804312f435e025dc: stopping tablet replica
I20260928 02:42:40.384874 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 26771
I20260928 02:42:40.385478 27208 tablet_replica.cc:330] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04: stopping tablet replica
I20260928 02:42:40.385582 27208 raft_consensus.cc:2229] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:42:40.385689 27208 raft_consensus.cc:2258] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:40.386088 27208 ts_tablet_manager.cc:1905] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:40.386336 27206 raft_consensus.cc:2258] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:40.386811 27206 ts_tablet_manager.cc:1905] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:40.387645 27208 ts_tablet_manager.cc:1918] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:40.387701 27208 log.cc:1196] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/691cd312ab2e433c8cc7f13df0fa5698
I20260928 02:42:40.387982 27208 ts_tablet_manager.cc:1939] T 691cd312ab2e433c8cc7f13df0fa5698 P b50b6587b4654cd693ba1a49a650aa04: Deleting consensus metadata
I20260928 02:42:40.388311 26695 catalog_manager.cc:4841] TS b50b6587b4654cd693ba1a49a650aa04 (127.9.130.3:36689): tablet 691cd312ab2e433c8cc7f13df0fa5698 (table test-table [id=c72cc332df6e48009c4b9725f42894c0]) successfully deleted
I20260928 02:42:40.389043 27206 ts_tablet_manager.cc:1918] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:40.389119 27206 log.cc:1196] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/691cd312ab2e433c8cc7f13df0fa5698
I20260928 02:42:40.389376 27206 ts_tablet_manager.cc:1939] T 691cd312ab2e433c8cc7f13df0fa5698 P 4166b56e8f45406f864f2a4f2e2bc2a1: Deleting consensus metadata
I20260928 02:42:40.389756 26694 catalog_manager.cc:4841] TS 4166b56e8f45406f864f2a4f2e2bc2a1 (127.9.130.2:46709): tablet 691cd312ab2e433c8cc7f13df0fa5698 (table test-table [id=c72cc332df6e48009c4b9725f42894c0]) successfully deleted
W20260928 02:42:40.391119 26695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:38949 (error 108)
I20260928 02:42:40.391392 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 26905
I20260928 02:42:40.391992 27176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.391469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:38949 (local address 127.0.0.1:53990)
0928 02:42:40.391914 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:40.391952 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424}
W20260928 02:42:40.392268 26695 catalog_manager.cc:4584] TS ee8c55492d09425d804312f435e025dc (127.9.130.1:38949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 691cd312ab2e433c8cc7f13df0fa5698: Network error: Client connection negotiation failed: client connection to 127.9.130.1:38949: connect: Connection refused (error 111)
I20260928 02:42:40.396697 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 27039
I20260928 02:42:40.401005 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 26677
2026-09-28T02:42:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2094 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T02:42:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:40Z Disabled control of system clock
I20260928 02:42:40.427505 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:35523
--webserver_interface=127.9.130.62
--webserver_port=0
--builtin_ntp_servers=127.9.130.20:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:35523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:42:40.509083 27213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:40.509280 27213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:40.509299 27213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:40.510435 27213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:40.510475 27213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:40.510488 27213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:40.510535 27213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:40.510552 27213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:40.512033 27213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:35523
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:35523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:40.512264 27213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:40.512548 27213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:40.517169 27221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:40.517159 27218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:40.517179 27219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:40.517346 27213 server_base.cc:1034] running on GCE node
I20260928 02:42:40.517673 27213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:40.517900 27213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:40.519070 27213 hybrid_clock.cc:648] HybridClock initialized: now 1790563360519040 us; error 44 us; skew 500 ppm
I20260928 02:42:40.520517 27213 webserver.cc:460] Webserver started at http://127.9.130.62:41871/ using document root <none> and password file <none>
I20260928 02:42:40.520792 27213 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:40.520884 27213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:40.521090 27213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:40.522666 27213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "4a4b3459e77246afa84a16f017209533"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "ac0256227923fa6646619da32d4d3f39"
server_key_iv: "87d090d1deba656e4fa3dd482ba5b4a9"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.523142 27213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "4a4b3459e77246afa84a16f017209533"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "ac0256227923fa6646619da32d4d3f39"
server_key_iv: "87d090d1deba656e4fa3dd482ba5b4a9"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.524873 27213 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:40.525822 27227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:40.526028 27213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:40.526121 27213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "4a4b3459e77246afa84a16f017209533"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "ac0256227923fa6646619da32d4d3f39"
server_key_iv: "87d090d1deba656e4fa3dd482ba5b4a9"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.526196 27213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:40.539567 27213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:40.539788 27213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:40.539943 27213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:40.543431 27213 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:35523
I20260928 02:42:40.543493 27279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:35523 every 8 connection(s)
I20260928 02:42:40.543788 27213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:40.544756 27280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:40.546952 27280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533: Bootstrap starting.
I20260928 02:42:40.547506 27280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:40.547803 27280 log.cc:824] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:40.548502 27280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533: No bootstrap required, opened a new log
I20260928 02:42:40.550038 27280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4b3459e77246afa84a16f017209533" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35523 } }
I20260928 02:42:40.550196 27280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:40.550242 27280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4b3459e77246afa84a16f017209533, State: Initialized, Role: FOLLOWER
I20260928 02:42:40.550391 27280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4b3459e77246afa84a16f017209533" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35523 } }
I20260928 02:42:40.550472 27280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:40.550514 27280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:40.550568 27280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:40.551059 27280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4b3459e77246afa84a16f017209533" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35523 } }
I20260928 02:42:40.551172 27280 leader_election.cc:302] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a4b3459e77246afa84a16f017209533; no voters:
I20260928 02:42:40.551345 27280 leader_election.cc:288] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:40.551393 27283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:40.551590 27280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:40.551644 27283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [term 1 LEADER]: Becoming Leader. State: Replica: 4a4b3459e77246afa84a16f017209533, State: Running, Role: LEADER
I20260928 02:42:40.551779 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 27213
I20260928 02:42:40.551767 27283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4b3459e77246afa84a16f017209533" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35523 } }
I20260928 02:42:40.551906 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:40.552119 9736 external_mini_cluster.cc:1428] Setting key 86287c085309d04c6c4bb78907671513
I20260928 02:42:40.552112 27284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a4b3459e77246afa84a16f017209533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4b3459e77246afa84a16f017209533" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35523 } } }
I20260928 02:42:40.552196 27284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:40.552313 27285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a4b3459e77246afa84a16f017209533. Latest consensus state: current_term: 1 leader_uuid: "4a4b3459e77246afa84a16f017209533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4b3459e77246afa84a16f017209533" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35523 } } }
I20260928 02:42:40.552410 27292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:40.552428 27285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:40.552928 27292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:40.554569 27292 catalog_manager.cc:1347] Generated new cluster ID: a9d54ee2ddac420180ffc41db5b89c1e
I20260928 02:42:40.554620 27292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:40.558316 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.552752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.9.130.62:35523)
0928 02:42:40.552941 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.552951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.552983 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:42:40.553115 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.553125 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.553203 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.553346 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:40.553665 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.554978 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.556090 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.556717 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.556844 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.557504 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:40.557508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:40.557509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:40.557722 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:40.557767 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:40.557783 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:40.557856 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.557924 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.558105 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260928 02:42:40.571712 27292 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:40.572220 27292 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:40.579216 27292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533: Generated new TSK 0
I20260928 02:42:40.579383 27292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:40.588435 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:35523
--builtin_ntp_servers=127.9.130.20:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:42:40.667428 27304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:40.667657 27304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:40.667693 27304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:40.668830 27304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:40.668895 27304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:40.668937 27304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:42:40.670269 27304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:35523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:40.670523 27304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:40.670806 27304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:40.676487 27309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:40.676488 27312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:40.676538 27310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:40.676831 27304 server_base.cc:1034] running on GCE node
I20260928 02:42:40.677057 27304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:40.677286 27304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:40.678454 27304 hybrid_clock.cc:648] HybridClock initialized: now 1790563360678444 us; error 37 us; skew 500 ppm
I20260928 02:42:40.679659 27304 webserver.cc:460] Webserver started at http://127.9.130.1:39181/ using document root <none> and password file <none>
I20260928 02:42:40.679953 27304 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:40.680038 27304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:40.680172 27304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:40.681196 27304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "214fb894572748f6a0fc50f68a704918"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "7f27a881a6171f20bffcb33ba38ab90d"
server_key_iv: "36da563972ab6e79dbcc03f72a0b8b68"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.681555 27304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "214fb894572748f6a0fc50f68a704918"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "7f27a881a6171f20bffcb33ba38ab90d"
server_key_iv: "36da563972ab6e79dbcc03f72a0b8b68"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.682793 27304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:42:40.683456 27318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:40.683629 27304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:42:40.683730 27304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "214fb894572748f6a0fc50f68a704918"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "7f27a881a6171f20bffcb33ba38ab90d"
server_key_iv: "36da563972ab6e79dbcc03f72a0b8b68"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.683815 27304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:40.694372 27304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:40.694625 27304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:40.694763 27304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:40.694967 27304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:40.695287 27304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:40.695345 27304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:40.695384 27304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:40.695410 27304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:40.701067 27304 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:44627
I20260928 02:42:40.701119 27431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:44627 every 8 connection(s)
I20260928 02:42:40.701409 27304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:42:40.702924 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 27304
I20260928 02:42:40.703052 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:42:40.703248 9736 external_mini_cluster.cc:1428] Setting key 550d82ab8c3d350a95d6991189a09327
I20260928 02:42:40.704246 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:35523
--builtin_ntp_servers=127.9.130.20:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:42:40.706864 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.702022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:59139 (local address 127.9.130.62:35523)
0928 02:42:40.702063 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.702065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.702293 (+ 228us) server_negotiation.cc:407] Connection header received
0928 02:42:40.702815 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.702816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.702835 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.702881 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:40.703752 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.704392 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.705521 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.705712 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.705769 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.706635 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:40.706636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:40.706636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:40.706640 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:40.706653 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:40.706657 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:40.706700 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.706784 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.706809 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:42:40.707137 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.702001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:35523 (local address 127.9.130.1:59139)
0928 02:42:40.702179 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:40.702209 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:40.702704 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:40.702959 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:40.702963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:40.702974 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:40.703661 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.703667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.704549 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.704550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.705421 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.705424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.705799 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.705800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.705959 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.706480 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:40.706480 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:40.706523 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:40.706552 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:40.706715 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:40.706719 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:40.706721 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:42:40.706788 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:42:40.706892 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260928 02:42:40.707621 27432 heartbeater.cc:347] Connected to a master server at 127.9.130.62:35523
I20260928 02:42:40.707728 27432 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:40.707929 27432 heartbeater.cc:511] Master 127.9.130.62:35523 requested a full tablet report, sending...
I20260928 02:42:40.708419 27244 ts_manager.cc:194] Registered new tserver with Master: 214fb894572748f6a0fc50f68a704918 (127.9.130.1:44627)
I20260928 02:42:40.709007 27244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.130.1:59139
W20260928 02:42:40.784045 27435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:40.784226 27435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:40.784246 27435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:40.785377 27435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:40.785415 27435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:40.785439 27435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:42:40.786824 27435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:35523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:40.787060 27435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:40.787374 27435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:40.792356 27443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:40.792517 27435 server_base.cc:1034] running on GCE node
W20260928 02:42:40.792366 27441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:40.792389 27440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:40.792806 27435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:40.792997 27435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:40.794135 27435 hybrid_clock.cc:648] HybridClock initialized: now 1790563360794105 us; error 46 us; skew 500 ppm
I20260928 02:42:40.795310 27435 webserver.cc:460] Webserver started at http://127.9.130.2:42221/ using document root <none> and password file <none>
I20260928 02:42:40.795547 27435 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:40.795591 27435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:40.795753 27435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:40.796795 27435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "df0391423f1d44a3b6bd493e688055d7"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "5de748846f68156d1196b5759fd6e7e0"
server_key_iv: "fce553e53f0b3c31ff839274d4172734"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.797156 27435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "df0391423f1d44a3b6bd493e688055d7"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "5de748846f68156d1196b5759fd6e7e0"
server_key_iv: "fce553e53f0b3c31ff839274d4172734"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.798388 27435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:42:40.799062 27449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:40.799249 27435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:40.799317 27435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "df0391423f1d44a3b6bd493e688055d7"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "5de748846f68156d1196b5759fd6e7e0"
server_key_iv: "fce553e53f0b3c31ff839274d4172734"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.799402 27435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:40.828419 27435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:40.828737 27435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:40.828892 27435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:40.829176 27435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:40.829558 27435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:40.829622 27435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:40.829664 27435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:40.829691 27435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:40.835640 27435 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:37107
I20260928 02:42:40.835680 27562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:37107 every 8 connection(s)
I20260928 02:42:40.836053 27435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:42:40.839308 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 27435
I20260928 02:42:40.839390 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:42:40.839633 9736 external_mini_cluster.cc:1428] Setting key 77cd62ae45423f473bbc9f5fb5fccdca
I20260928 02:42:40.840585 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:35523
--builtin_ntp_servers=127.9.130.20:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:42:40.841754 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.836798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:48469 (local address 127.9.130.62:35523)
0928 02:42:40.836870 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.836872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.837121 (+ 249us) server_negotiation.cc:407] Connection header received
0928 02:42:40.837652 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.837654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.837666 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.837716 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:40.838380 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.838886 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.840140 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.840431 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.840518 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.841476 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:40.841477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:40.841477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:40.841481 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:40.841496 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:40.841500 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:40.841571 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.841647 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.841678 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:42:40.842002 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.836804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:35523 (local address 127.9.130.2:48469)
0928 02:42:40.837000 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:40.837036 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:40.837541 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:40.837786 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:40.837790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:40.837798 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:40.838286 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.838290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.838985 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.838986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.840055 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.840058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.840526 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.840528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.840688 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.841308 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:40.841309 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:40.841344 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:40.841368 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:40.841571 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:40.841577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:40.841580 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:40.841652 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:42:40.841753 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260928 02:42:40.842521 27563 heartbeater.cc:347] Connected to a master server at 127.9.130.62:35523
I20260928 02:42:40.842624 27563 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:40.842800 27563 heartbeater.cc:511] Master 127.9.130.62:35523 requested a full tablet report, sending...
I20260928 02:42:40.843179 27244 ts_manager.cc:194] Registered new tserver with Master: df0391423f1d44a3b6bd493e688055d7 (127.9.130.2:37107)
I20260928 02:42:40.843578 27244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.130.2:48469
W20260928 02:42:40.920433 27566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:40.920647 27566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:40.920684 27566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:40.921846 27566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:40.921904 27566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:40.921947 27566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:42:40.923277 27566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:35523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:40.923534 27566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:40.923831 27566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:40.928695 27572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:40.928797 27571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:40.928697 27574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:40.928929 27566 server_base.cc:1034] running on GCE node
I20260928 02:42:40.929366 27566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:40.929641 27566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:40.930835 27566 hybrid_clock.cc:648] HybridClock initialized: now 1790563360930815 us; error 46 us; skew 500 ppm
I20260928 02:42:40.932211 27566 webserver.cc:460] Webserver started at http://127.9.130.3:45059/ using document root <none> and password file <none>
I20260928 02:42:40.932497 27566 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:40.932580 27566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:40.932754 27566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:40.933910 27566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "65235d3a31e24758875267d98ed2b06c"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "810428efbd5c241d5bf36f7955b12bd0"
server_key_iv: "5385dd57482de4cf8154739123e48100"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.934329 27566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "65235d3a31e24758875267d98ed2b06c"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "810428efbd5c241d5bf36f7955b12bd0"
server_key_iv: "5385dd57482de4cf8154739123e48100"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.935775 27566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:42:40.936604 27580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:40.936776 27566 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:40.936894 27566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "65235d3a31e24758875267d98ed2b06c"
format_stamp: "Formatted at 2026-09-28 02:42:40 on dist-test-slave-kkrh"
server_key: "810428efbd5c241d5bf36f7955b12bd0"
server_key_iv: "5385dd57482de4cf8154739123e48100"
server_key_version: "encryptionkey@0"
I20260928 02:42:40.936985 27566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:40.949815 27566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:40.950119 27566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:40.950285 27566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:40.950524 27566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:40.950889 27566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:40.950960 27566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:40.951009 27566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:40.951058 27566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:40.957934 27566 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:39443
I20260928 02:42:40.957976 27693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:39443 every 8 connection(s)
I20260928 02:42:40.958333 27566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:42:40.963933 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.959161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:43659 (local address 127.9.130.62:35523)
0928 02:42:40.959253 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.959256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.959514 (+ 258us) server_negotiation.cc:407] Connection header received
0928 02:42:40.960094 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.960095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.960105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.960171 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:40.960872 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.961441 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.962534 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.962721 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.962779 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.963600 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:40.963602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:40.963602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:40.963606 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:40.963619 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:40.963623 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:40.963667 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.963752 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.963775 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:42:40.964038 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.959151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:35523 (local address 127.9.130.3:43659)
0928 02:42:40.959384 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:40.959417 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:40.959972 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:40.960238 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:40.960241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:40.960249 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:40.960767 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.960771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.961545 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.961546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.962445 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.962448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.962797 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.962798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.962900 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.963447 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:40.963447 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:40.963497 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:40.963518 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:40.963682 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:40.963685 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:40.963688 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:40.963738 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:42:40.963854 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":118,"threads_started":1}
I20260928 02:42:40.964556 27694 heartbeater.cc:347] Connected to a master server at 127.9.130.62:35523
I20260928 02:42:40.964648 27694 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:40.964814 27694 heartbeater.cc:511] Master 127.9.130.62:35523 requested a full tablet report, sending...
I20260928 02:42:40.965167 27244 ts_manager.cc:194] Registered new tserver with Master: 65235d3a31e24758875267d98ed2b06c (127.9.130.3:39443)
I20260928 02:42:40.965509 27244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.130.3:43659
I20260928 02:42:40.965998 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 27566
I20260928 02:42:40.966076 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:42:40.966265 9736 external_mini_cluster.cc:1428] Setting key ab2e02c597760e3771d945537f9b01fa
I20260928 02:42:40.967126 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:42:40.971328 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.968518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.9.130.62:35523)
0928 02:42:40.968595 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.968597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.968689 (+ 92us) server_negotiation.cc:407] Connection header received
0928 02:42:40.968764 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.968765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.968780 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.968835 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:40.969011 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.969495 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.970249 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.970434 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.970504 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.971088 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:40.971089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:40.971090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:40.971094 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:40.971105 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:40.971109 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:40.971171 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.971231 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.971251 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:42:40.972740 27244 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:42:40.974185 27244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:42:40.980362 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.976386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:37107 (local address 127.0.0.1:50172)
0928 02:42:40.976641 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:40.976652 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:40.976708 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:40.976968 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:40.976971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:40.976974 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:40.977039 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.977041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.978042 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.978043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.979149 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.979151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.980079 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.980080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.980157 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.980170 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:42:40.980243 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:42:40.980292 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":92,"threads_started":1}
I20260928 02:42:40.980379 27704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.976477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.9.130.2:37107)
0928 02:42:40.976712 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.976717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.976730 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:42:40.976782 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.976783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.976797 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.976896 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:40.977113 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.977954 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.979236 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.979997 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.980080 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.980167 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.980258 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.980292 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":75,"threads_started":1}
I20260928 02:42:40.980549 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.976366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:39443 (local address 127.0.0.1:33224)
0928 02:42:40.976530 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:40.976568 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:40.976630 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:40.977169 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:40.977170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:40.977172 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:40.977209 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.977211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.978495 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.978496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.979447 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.979449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.980316 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.980317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.980394 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.980398 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:42:40.980429 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 02:42:40.980468 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260928 02:42:40.980871 27706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.976582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:44627 (local address 127.0.0.1:51144)
0928 02:42:40.976788 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:40.976798 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:40.976862 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:40.977293 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:40.977294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:40.977296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:40.977349 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.977351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.978587 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.978588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.979666 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:40.979669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.980639 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.980640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:40.980710 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.980713 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:40.980760 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:42:40.980792 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":67,"threads_started":1}
I20260928 02:42:40.980969 27705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.976539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.9.130.3:39443)
0928 02:42:40.977017 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.977022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.977031 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:42:40.977063 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.977065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.977080 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.977284 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:40.977322 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.978405 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.979538 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.980214 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.980719 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.980806 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.980830 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.980867 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":73,"threads_started":1}
I20260928 02:42:40.981134 27707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:40.976769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.9.130.1:44627)
0928 02:42:40.976955 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:40.976960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:40.976969 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:42:40.977018 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:40.977020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:40.977035 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:40.977224 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:40.977436 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.978507 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.979755 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:40.980462 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:40.980521 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:40.980604 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:40.981031 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 02:42:40.981077 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
I20260928 02:42:40.981262 27497 tablet_service.cc:1467] Processing CreateTablet for tablet c6d0d630efc24bf09c65ca6525040992 (DEFAULT_TABLE table=test-table [id=94e47d0e04d6454fae734cd973941ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:40.981632 27497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d0d630efc24bf09c65ca6525040992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:40.981736 27366 tablet_service.cc:1467] Processing CreateTablet for tablet c6d0d630efc24bf09c65ca6525040992 (DEFAULT_TABLE table=test-table [id=94e47d0e04d6454fae734cd973941ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:40.982074 27366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d0d630efc24bf09c65ca6525040992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:40.984289 27708 tablet_bootstrap.cc:492] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7: Bootstrap starting.
I20260928 02:42:40.984815 27628 tablet_service.cc:1467] Processing CreateTablet for tablet c6d0d630efc24bf09c65ca6525040992 (DEFAULT_TABLE table=test-table [id=94e47d0e04d6454fae734cd973941ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:40.984938 27708 tablet_bootstrap.cc:654] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:40.985113 27628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d0d630efc24bf09c65ca6525040992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:40.985255 27708 log.cc:824] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:40.985988 27708 tablet_bootstrap.cc:492] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7: No bootstrap required, opened a new log
I20260928 02:42:40.986055 27708 ts_tablet_manager.cc:1397] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:42:40.987671 27711 tablet_bootstrap.cc:492] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c: Bootstrap starting.
I20260928 02:42:40.987987 27709 tablet_bootstrap.cc:492] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918: Bootstrap starting.
I20260928 02:42:40.988210 27708 raft_consensus.cc:348] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65235d3a31e24758875267d98ed2b06c" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39443 } } peers { permanent_uuid: "df0391423f1d44a3b6bd493e688055d7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37107 } } peers { permanent_uuid: "214fb894572748f6a0fc50f68a704918" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 44627 } }
I20260928 02:42:40.988358 27708 raft_consensus.cc:374] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:40.988399 27711 tablet_bootstrap.cc:654] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:40.988399 27708 raft_consensus.cc:729] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0391423f1d44a3b6bd493e688055d7, State: Initialized, Role: FOLLOWER
I20260928 02:42:40.988529 27708 consensus_queue.cc:260] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65235d3a31e24758875267d98ed2b06c" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39443 } } peers { permanent_uuid: "df0391423f1d44a3b6bd493e688055d7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37107 } } peers { permanent_uuid: "214fb894572748f6a0fc50f68a704918" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 44627 } }
I20260928 02:42:40.988698 27711 log.cc:824] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:40.988816 27708 ts_tablet_manager.cc:1428] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 02:42:40.988643 27709 tablet_bootstrap.cc:654] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:40.988989 27563 heartbeater.cc:503] Master 127.9.130.62:35523 was elected leader, sending a full tablet report...
I20260928 02:42:40.989176 27709 log.cc:824] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:40.989892 27709 tablet_bootstrap.cc:492] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918: No bootstrap required, opened a new log
I20260928 02:42:40.989964 27709 ts_tablet_manager.cc:1397] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:42:40.989998 27711 tablet_bootstrap.cc:492] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c: No bootstrap required, opened a new log
I20260928 02:42:40.990069 27711 ts_tablet_manager.cc:1397] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:42:40.991772 27711 raft_consensus.cc:348] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65235d3a31e24758875267d98ed2b06c" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39443 } } peers { permanent_uuid: "df0391423f1d44a3b6bd493e688055d7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37107 } } peers { permanent_uuid: "214fb894572748f6a0fc50f68a704918" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 44627 } }
I20260928 02:42:40.991886 27709 raft_consensus.cc:348] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65235d3a31e24758875267d98ed2b06c" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39443 } } peers { permanent_uuid: "df0391423f1d44a3b6bd493e688055d7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37107 } } peers { permanent_uuid: "214fb894572748f6a0fc50f68a704918" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 44627 } }
I20260928 02:42:40.991993 27711 raft_consensus.cc:374] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:40.992044 27709 raft_consensus.cc:374] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:40.992041 27711 raft_consensus.cc:729] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65235d3a31e24758875267d98ed2b06c, State: Initialized, Role: FOLLOWER
I20260928 02:42:40.992089 27709 raft_consensus.cc:729] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214fb894572748f6a0fc50f68a704918, State: Initialized, Role: FOLLOWER
I20260928 02:42:40.992164 27711 consensus_queue.cc:260] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65235d3a31e24758875267d98ed2b06c" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39443 } } peers { permanent_uuid: "df0391423f1d44a3b6bd493e688055d7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37107 } } peers { permanent_uuid: "214fb894572748f6a0fc50f68a704918" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 44627 } }
I20260928 02:42:40.992205 27709 consensus_queue.cc:260] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65235d3a31e24758875267d98ed2b06c" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39443 } } peers { permanent_uuid: "df0391423f1d44a3b6bd493e688055d7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37107 } } peers { permanent_uuid: "214fb894572748f6a0fc50f68a704918" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 44627 } }
I20260928 02:42:40.992440 27711 ts_tablet_manager.cc:1428] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:42:40.992471 27709 ts_tablet_manager.cc:1428] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:42:40.992516 27432 heartbeater.cc:503] Master 127.9.130.62:35523 was elected leader, sending a full tablet report...
I20260928 02:42:40.992493 27694 heartbeater.cc:503] Master 127.9.130.62:35523 was elected leader, sending a full tablet report...
W20260928 02:42:41.086629 27564 tablet.cc:2369] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:42:41.202114 27433 tablet.cc:2369] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:42:41.208937 27695 tablet.cc:2369] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:42:41.260972 27716 raft_consensus.cc:482] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:42:41.261089 27716 raft_consensus.cc:504] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65235d3a31e24758875267d98ed2b06c" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39443 } } peers { permanent_uuid: "df0391423f1d44a3b6bd493e688055d7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37107 } } peers { permanent_uuid: "214fb894572748f6a0fc50f68a704918" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 44627 } }
I20260928 02:42:41.261505 27716 leader_election.cc:288] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65235d3a31e24758875267d98ed2b06c (127.9.130.3:39443), df0391423f1d44a3b6bd493e688055d7 (127.9.130.2:37107)
I20260928 02:42:41.264644 27704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:41.261825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:43029 (local address 127.9.130.2:37107)
0928 02:42:41.261948 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:41.261950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:41.261958 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:42:41.261994 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:41.261995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:41.262002 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:41.262070 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:41.262238 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:41.262804 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:41.263916 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:41.264274 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:41.264369 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:41.264379 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:41.264536 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:42:41.264571 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 02:42:41.264659 27717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:41.261681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:37107 (local address 127.9.130.1:43029)
0928 02:42:41.261808 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:41.261821 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:41.261879 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:41.262075 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:41.262076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:41.262078 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:41.262148 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:41.262149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:41.262914 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:41.262915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:41.263767 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:41.263769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:41.264370 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:41.264371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:41.264448 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:41.264452 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:42:41.264530 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:42:41.264571 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":64,"thread_start_us":64,"threads_started":1}
I20260928 02:42:41.264950 27705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:41.261889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:54577 (local address 127.9.130.3:39443)
0928 02:42:41.261939 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:41.261941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:41.261954 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:42:41.262073 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:41.262074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:41.262083 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:41.262159 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:41.262346 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:41.262955 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:41.264064 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:41.264485 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:41.264583 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:41.264592 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:41.264803 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 02:42:41.264864 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:42:41.265084 27517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d0d630efc24bf09c65ca6525040992" candidate_uuid: "214fb894572748f6a0fc50f68a704918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0391423f1d44a3b6bd493e688055d7" is_pre_election: true
I20260928 02:42:41.265168 27718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:41.261729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:39443 (local address 127.9.130.1:54577)
0928 02:42:41.261893 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:41.261905 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:41.261970 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:41.262162 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:41.262163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:41.262165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:41.262253 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:41.262255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:41.263052 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:41.263053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:41.263973 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:41.263975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:41.264586 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:41.264587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:41.264673 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:41.264676 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:41.265046 (+ 370us) client_negotiation.cc:240] Negotiation successful
0928 02:42:41.265091 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":1,"thread_start_us":120,"threads_started":1}
I20260928 02:42:41.265266 27517 raft_consensus.cc:2454] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214fb894572748f6a0fc50f68a704918 in term 0.
I20260928 02:42:41.265534 27321 leader_election.cc:302] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214fb894572748f6a0fc50f68a704918, df0391423f1d44a3b6bd493e688055d7; no voters:
I20260928 02:42:41.265681 27716 raft_consensus.cc:2781] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:42:41.265667 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d0d630efc24bf09c65ca6525040992" candidate_uuid: "214fb894572748f6a0fc50f68a704918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65235d3a31e24758875267d98ed2b06c" is_pre_election: true
I20260928 02:42:41.265753 27716 raft_consensus.cc:482] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:42:41.265776 27716 raft_consensus.cc:3037] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:41.265810 27648 raft_consensus.cc:2454] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214fb894572748f6a0fc50f68a704918 in term 0.
I20260928 02:42:41.266517 27716 raft_consensus.cc:504] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65235d3a31e24758875267d98ed2b06c" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39443 } } peers { permanent_uuid: "df0391423f1d44a3b6bd493e688055d7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37107 } } peers { permanent_uuid: "214fb894572748f6a0fc50f68a704918" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 44627 } }
I20260928 02:42:41.266659 27716 leader_election.cc:288] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [CANDIDATE]: Term 1 election: Requested vote from peers 65235d3a31e24758875267d98ed2b06c (127.9.130.3:39443), df0391423f1d44a3b6bd493e688055d7 (127.9.130.2:37107)
I20260928 02:42:41.266816 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d0d630efc24bf09c65ca6525040992" candidate_uuid: "214fb894572748f6a0fc50f68a704918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65235d3a31e24758875267d98ed2b06c"
I20260928 02:42:41.266855 27517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d0d630efc24bf09c65ca6525040992" candidate_uuid: "214fb894572748f6a0fc50f68a704918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0391423f1d44a3b6bd493e688055d7"
I20260928 02:42:41.266898 27648 raft_consensus.cc:3037] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:41.266953 27517 raft_consensus.cc:3037] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:41.267668 27648 raft_consensus.cc:2454] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214fb894572748f6a0fc50f68a704918 in term 1.
I20260928 02:42:41.267874 27517 raft_consensus.cc:2454] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214fb894572748f6a0fc50f68a704918 in term 1.
I20260928 02:42:41.267963 27320 leader_election.cc:302] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214fb894572748f6a0fc50f68a704918, 65235d3a31e24758875267d98ed2b06c; no voters:
I20260928 02:42:41.268061 27716 raft_consensus.cc:2781] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:41.268232 27716 raft_consensus.cc:686] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [term 1 LEADER]: Becoming Leader. State: Replica: 214fb894572748f6a0fc50f68a704918, State: Running, Role: LEADER
I20260928 02:42:41.268325 27716 consensus_queue.cc:237] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65235d3a31e24758875267d98ed2b06c" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39443 } } peers { permanent_uuid: "df0391423f1d44a3b6bd493e688055d7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37107 } } peers { permanent_uuid: "214fb894572748f6a0fc50f68a704918" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 44627 } }
I20260928 02:42:41.269160 27244 catalog_manager.cc:5495] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 reported cstate change: term changed from 0 to 1, leader changed from <none> to 214fb894572748f6a0fc50f68a704918 (127.9.130.1). New cstate: current_term: 1 leader_uuid: "214fb894572748f6a0fc50f68a704918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65235d3a31e24758875267d98ed2b06c" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df0391423f1d44a3b6bd493e688055d7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "214fb894572748f6a0fc50f68a704918" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 44627 } health_report { overall_health: HEALTHY } } }
I20260928 02:42:41.332731 27707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:41.330058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51148 (local address 127.9.130.1:44627)
0928 02:42:41.330120 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:41.330123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:41.330133 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:42:41.330297 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:41.330299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:41.330340 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:41.330433 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:42:41.330578 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:41.331070 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:41.331910 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:41.332109 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:41.332210 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:41.332380 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:42:41.332491 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:42:41.332595 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:41.332623 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 02:42:41.332644 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:42:41.334645 27648 raft_consensus.cc:1261] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c [term 1 FOLLOWER]: Refusing update from remote peer 214fb894572748f6a0fc50f68a704918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:41.334645 27517 raft_consensus.cc:1261] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7 [term 1 FOLLOWER]: Refusing update from remote peer 214fb894572748f6a0fc50f68a704918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:41.335055 27719 consensus_queue.cc:1035] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0391423f1d44a3b6bd493e688055d7" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:41.335320 27716 consensus_queue.cc:1035] T c6d0d630efc24bf09c65ca6525040992 P 214fb894572748f6a0fc50f68a704918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65235d3a31e24758875267d98ed2b06c" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 39443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:41.337813 27722 mvcc.cc:205] Tried to move back new op lower bound from 7334147528023957504 to 7334147527755747328. Current Snapshot: MvccSnapshot[applied={T|T < 7334147528023957504}]
I20260928 02:42:41.338191 27723 mvcc.cc:205] Tried to move back new op lower bound from 7334147528023957504 to 7334147527755747328. Current Snapshot: MvccSnapshot[applied={T|T < 7334147528023957504}]
I20260928 02:42:41.340452 27244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:41.340531 27244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:41.341321 27244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a4b3459e77246afa84a16f017209533: Sending DeleteTablet for 3 replicas of tablet c6d0d630efc24bf09c65ca6525040992
I20260928 02:42:41.341651 27628 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d0d630efc24bf09c65ca6525040992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:41 UTC) from {username='slave'} at 127.0.0.1:33224
I20260928 02:42:41.341739 27497 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d0d630efc24bf09c65ca6525040992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:41 UTC) from {username='slave'} at 127.0.0.1:50172
I20260928 02:42:41.341741 27366 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d0d630efc24bf09c65ca6525040992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:41 UTC) from {username='slave'} at 127.0.0.1:51144
I20260928 02:42:41.341950 27732 tablet_replica.cc:330] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7: stopping tablet replica
I20260928 02:42:41.342044 27732 raft_consensus.cc:2229] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:42:41.342142 27732 raft_consensus.cc:2258] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:41.342154 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 27304
I20260928 02:42:41.342949 27732 ts_tablet_manager.cc:1905] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:41.345110 27732 ts_tablet_manager.cc:1918] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:41.345187 27732 log.cc:1196] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/c6d0d630efc24bf09c65ca6525040992
I20260928 02:42:41.345436 27732 ts_tablet_manager.cc:1939] T c6d0d630efc24bf09c65ca6525040992 P df0391423f1d44a3b6bd493e688055d7: Deleting consensus metadata
I20260928 02:42:41.345861 27230 catalog_manager.cc:4841] TS df0391423f1d44a3b6bd493e688055d7 (127.9.130.2:37107): tablet c6d0d630efc24bf09c65ca6525040992 (table test-table [id=94e47d0e04d6454fae734cd973941ba7]) successfully deleted
W20260928 02:42:41.348837 27230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:44627 (error 108)
I20260928 02:42:41.348939 27731 tablet_replica.cc:330] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c: stopping tablet replica
I20260928 02:42:41.349046 27731 raft_consensus.cc:2229] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:42:41.349086 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 27435
I20260928 02:42:41.349157 27731 raft_consensus.cc:2258] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:41.349752 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:41.349200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:44627 (local address 127.0.0.1:51158)
0928 02:42:41.349665 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:41.349713 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442}
W20260928 02:42:41.350186 27230 catalog_manager.cc:4584] TS 214fb894572748f6a0fc50f68a704918 (127.9.130.1:44627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6d0d630efc24bf09c65ca6525040992: Network error: Client connection negotiation failed: client connection to 127.9.130.1:44627: connect: Connection refused (error 111)
I20260928 02:42:41.350711 27731 ts_tablet_manager.cc:1905] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:41.352051 27731 ts_tablet_manager.cc:1918] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:41.352124 27731 log.cc:1196] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/c6d0d630efc24bf09c65ca6525040992
I20260928 02:42:41.352356 27731 ts_tablet_manager.cc:1939] T c6d0d630efc24bf09c65ca6525040992 P 65235d3a31e24758875267d98ed2b06c: Deleting consensus metadata
I20260928 02:42:41.352851 27229 catalog_manager.cc:4841] TS 65235d3a31e24758875267d98ed2b06c (127.9.130.3:39443): tablet c6d0d630efc24bf09c65ca6525040992 (table test-table [id=94e47d0e04d6454fae734cd973941ba7]) successfully deleted
I20260928 02:42:41.354007 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 27566
I20260928 02:42:41.358425 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 27213
2026-09-28T02:42:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (957 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:41 dist-test-slave-kkrh krb5kdc[27737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:41 dist-test-slave-kkrh krb5kdc[27737](info): set up 2 sockets
Sep 28 02:42:41 dist-test-slave-kkrh krb5kdc[27737](info): commencing operation
krb5kdc: starting...
W20260928 02:42:42.379606 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:42:42 dist-test-slave-kkrh krb5kdc[27737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:42:42.491667 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:42155
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:42.570708 27750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:42.570897 27750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:42.570919 27750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:42.571985 27750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:42.572021 27750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:42.572034 27750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:42.572047 27750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:42.572059 27750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:42.573449 27750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:42155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:42155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:42.573633 27750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:42.573794 27750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:42.578547 27755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:42.578611 27758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:42.578727 27750 server_base.cc:1034] running on GCE node
W20260928 02:42:42.578807 27756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:42.579097 27750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:42.579398 27750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:42.580576 27750 hybrid_clock.cc:648] HybridClock initialized: now 1790563362580548 us; error 38 us; skew 500 ppm
Sep 28 02:42:42 dist-test-slave-kkrh krb5kdc[27737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563362, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:42.591130 27750 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:42:42.591643 27750 webserver.cc:460] Webserver started at http://127.9.130.62:37973/ using document root <none> and password file <none>
I20260928 02:42:42.591921 27750 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:42.591989 27750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:42.592136 27750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:42.593199 27750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "06129bde60234c5eab8cf03aac30c816"
format_stamp: "Formatted at 2026-09-28 02:42:42 on dist-test-slave-kkrh"
server_key: "ea943f19b3a3f4605bfbbc71edb0afdc"
server_key_iv: "ad36d12263eaffbdae26c6b117d78253"
server_key_version: "encryptionkey@0"
I20260928 02:42:42.593569 27750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "06129bde60234c5eab8cf03aac30c816"
format_stamp: "Formatted at 2026-09-28 02:42:42 on dist-test-slave-kkrh"
server_key: "ea943f19b3a3f4605bfbbc71edb0afdc"
server_key_iv: "ad36d12263eaffbdae26c6b117d78253"
server_key_version: "encryptionkey@0"
I20260928 02:42:42.594888 27750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:42:42.595635 27765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:42.595875 27750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:42.595955 27750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "06129bde60234c5eab8cf03aac30c816"
format_stamp: "Formatted at 2026-09-28 02:42:42 on dist-test-slave-kkrh"
server_key: "ea943f19b3a3f4605bfbbc71edb0afdc"
server_key_iv: "ad36d12263eaffbdae26c6b117d78253"
server_key_version: "encryptionkey@0"
I20260928 02:42:42.596030 27750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:42.619314 27750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:42.621148 27750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:42.621299 27750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:42.625355 27750 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:42155
I20260928 02:42:42.625392 27817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:42155 every 8 connection(s)
I20260928 02:42:42.625798 27750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:42.626049 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 27750
I20260928 02:42:42.626160 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:42.626372 9736 external_mini_cluster.cc:1428] Setting key c0be15339989de4a71d1965bc79a85f6
I20260928 02:42:42.626753 27818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:42.629475 27818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816: Bootstrap starting.
I20260928 02:42:42.630100 27818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:42.630373 27818 log.cc:824] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:42.631013 27818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816: No bootstrap required, opened a new log
I20260928 02:42:42.632944 27818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06129bde60234c5eab8cf03aac30c816" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42155 } }
I20260928 02:42:42.633108 27818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:42.633181 27818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06129bde60234c5eab8cf03aac30c816, State: Initialized, Role: FOLLOWER
Sep 28 02:42:42 dist-test-slave-kkrh krb5kdc[27737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:42.633352 27818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06129bde60234c5eab8cf03aac30c816" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42155 } }
I20260928 02:42:42.633436 27818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:42.633479 27818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:42.633529 27818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:42.634025 27818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06129bde60234c5eab8cf03aac30c816" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42155 } }
I20260928 02:42:42.634157 27818 leader_election.cc:302] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06129bde60234c5eab8cf03aac30c816; no voters:
I20260928 02:42:42.634320 27818 leader_election.cc:288] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:42.634359 27823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:42.634500 27823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [term 1 LEADER]: Becoming Leader. State: Replica: 06129bde60234c5eab8cf03aac30c816, State: Running, Role: LEADER
I20260928 02:42:42.634606 27818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:42.634604 27823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06129bde60234c5eab8cf03aac30c816" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42155 } }
I20260928 02:42:42.634991 27823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06129bde60234c5eab8cf03aac30c816. Latest consensus state: current_term: 1 leader_uuid: "06129bde60234c5eab8cf03aac30c816" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06129bde60234c5eab8cf03aac30c816" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42155 } } }
I20260928 02:42:42.635071 27823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:42.635061 27825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06129bde60234c5eab8cf03aac30c816" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06129bde60234c5eab8cf03aac30c816" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 42155 } } }
I20260928 02:42:42.635339 27825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:42.635402 27831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:42.635950 27831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:42.637406 27840 mvcc.cc:205] Tried to move back new op lower bound from 7334147533360979968 to 7334147533351956480. Current Snapshot: MvccSnapshot[applied={T|T < 7334147533360979968}]
I20260928 02:42:42.637688 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:42.626893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.9.130.62:42155)
0928 02:42:42.627124 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:42.627133 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:42.627167 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:42:42.628133 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:42.628140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:42.628192 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:42.628377 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:42.629019 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:42.629951 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:42.630836 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:42.631273 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:42.631386 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:42.633828 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:42.633832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:42.633832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:42.633844 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:42.635975 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:42.636496 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:42.636500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:42.636501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:42.636603 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:42.636805 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:42.636806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:42.636806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:42.637043 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:42.637192 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:42.637293 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 02:42:42.637522 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
I20260928 02:42:42.637846 27831 catalog_manager.cc:1347] Generated new cluster ID: 2db4f345e18a4fd8934056c8de9e6d76
I20260928 02:42:42.637892 27831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:42.663040 27831 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:42.663628 27831 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:42.669222 27831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816: Generated new TSK 0
I20260928 02:42:42.669447 27831 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:42:42.699278 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42155
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:42.782217 27844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:42.782402 27844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:42.782425 27844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:42.783527 27844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:42.783563 27844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:42.783588 27844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:42:42.785058 27844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:42.785238 27844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:42.785418 27844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:42.790356 27849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:42.790378 27852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:42.790443 27844 server_base.cc:1034] running on GCE node
W20260928 02:42:42.790377 27850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:42.790804 27844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:42.791093 27844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:42.792295 27844 hybrid_clock.cc:648] HybridClock initialized: now 1790563362792290 us; error 36 us; skew 500 ppm
Sep 28 02:42:42 dist-test-slave-kkrh krb5kdc[27737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563362, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:42.795063 27844 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:42:42.795558 27844 webserver.cc:460] Webserver started at http://127.9.130.1:39079/ using document root <none> and password file <none>
I20260928 02:42:42.795792 27844 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:42.795894 27844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:42.796051 27844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:42.797070 27844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "37ed372cfc6f4fd084bc8efe92186820"
format_stamp: "Formatted at 2026-09-28 02:42:42 on dist-test-slave-kkrh"
server_key: "b7bad8d804202613a7559699b316917d"
server_key_iv: "76f45bb251e4aef21a6fc9a5d327e2e3"
server_key_version: "encryptionkey@0"
I20260928 02:42:42.797431 27844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "37ed372cfc6f4fd084bc8efe92186820"
format_stamp: "Formatted at 2026-09-28 02:42:42 on dist-test-slave-kkrh"
server_key: "b7bad8d804202613a7559699b316917d"
server_key_iv: "76f45bb251e4aef21a6fc9a5d327e2e3"
server_key_version: "encryptionkey@0"
I20260928 02:42:42.798776 27844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:42:42.799515 27859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:42.799692 27844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:42.799791 27844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "37ed372cfc6f4fd084bc8efe92186820"
format_stamp: "Formatted at 2026-09-28 02:42:42 on dist-test-slave-kkrh"
server_key: "b7bad8d804202613a7559699b316917d"
server_key_iv: "76f45bb251e4aef21a6fc9a5d327e2e3"
server_key_version: "encryptionkey@0"
I20260928 02:42:42.799885 27844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:42.807996 27844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:42.809757 27844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:42.809913 27844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:42.810141 27844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:42.810452 27844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:42.810513 27844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:42.810549 27844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:42.810576 27844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:42.816560 27844 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:46159
I20260928 02:42:42.816599 27972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:46159 every 8 connection(s)
I20260928 02:42:42.816921 27844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:42:42 dist-test-slave-kkrh krb5kdc[27737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563362, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:42.824363 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 27844
I20260928 02:42:42.824483 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:42:42.824726 9736 external_mini_cluster.cc:1428] Setting key 9d90f2f22e0a0c398d7fbcb3993cbb57
I20260928 02:42:42.826732 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:42.817691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:56859 (local address 127.9.130.62:42155)
0928 02:42:42.817776 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:42.817778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:42.817950 (+ 172us) server_negotiation.cc:407] Connection header received
0928 02:42:42.818465 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:42.818466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:42.818479 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:42.818541 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:42.819466 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:42.820058 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:42.821058 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:42.821278 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:42.821365 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:42.823547 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:42.823548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:42.823549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:42.823555 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:42.825094 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:42.825639 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:42.825640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:42.825641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:42.825705 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:42.826011 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:42.826012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:42.826012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:42.826163 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:42.826233 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:42.826491 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 02:42:42.826590 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:42:42.826804 27975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:42.817647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:42155 (local address 127.9.130.1:56859)
0928 02:42:42.817822 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:42.817870 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:42.818361 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:42.818617 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:42.818621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:42.819183 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:42.819361 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:42.819366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:42.820151 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:42.820152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:42.820947 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:42.820949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:42.821375 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:42.821376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:42.821509 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:42.821997 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:42.822029 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:42.823459 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:42.825156 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:42.825163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:42.825165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:42.825552 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:42.825808 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:42.825809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:42.825810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:42.825926 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:42.826234 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:42.826239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:42.826361 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 02:42:42.826459 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:42:42.826589 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260928 02:42:42.827405 27973 heartbeater.cc:347] Connected to a master server at 127.9.130.62:42155
I20260928 02:42:42.827507 27973 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:42.827715 27973 heartbeater.cc:511] Master 127.9.130.62:42155 requested a full tablet report, sending...
I20260928 02:42:42.828286 27782 ts_manager.cc:194] Registered new tserver with Master: 37ed372cfc6f4fd084bc8efe92186820 (127.9.130.1:46159)
I20260928 02:42:42.828958 27782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:56859
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:42:42.850409 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42155
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:42.929963 27978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:42.930150 27978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:42.930171 27978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:42.931218 27978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:42.931252 27978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:42.931277 27978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:42:42.932696 27978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:42.932878 27978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:42.933034 27978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:42.938237 27983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:42.938205 27986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:42.938344 27978 server_base.cc:1034] running on GCE node
W20260928 02:42:42.938210 27984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:42.938720 27978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:42.938962 27978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:42.940119 27978 hybrid_clock.cc:648] HybridClock initialized: now 1790563362940098 us; error 34 us; skew 500 ppm
Sep 28 02:42:42 dist-test-slave-kkrh krb5kdc[27737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563362, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:42.942530 27978 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:42:42.942939 27978 webserver.cc:460] Webserver started at http://127.9.130.2:42571/ using document root <none> and password file <none>
I20260928 02:42:42.943169 27978 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:42.943236 27978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:42.943383 27978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:42.944489 27978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "bcf8d79c95544039b2af5aaee160fdca"
format_stamp: "Formatted at 2026-09-28 02:42:42 on dist-test-slave-kkrh"
server_key: "730268f9c96eccbd1c04c783fb970cad"
server_key_iv: "750e041e43d5c279ec57708b72e2df78"
server_key_version: "encryptionkey@0"
I20260928 02:42:42.944871 27978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "bcf8d79c95544039b2af5aaee160fdca"
format_stamp: "Formatted at 2026-09-28 02:42:42 on dist-test-slave-kkrh"
server_key: "730268f9c96eccbd1c04c783fb970cad"
server_key_iv: "750e041e43d5c279ec57708b72e2df78"
server_key_version: "encryptionkey@0"
I20260928 02:42:42.946197 27978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:42:42.946950 27993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:42.947149 27978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:42.947254 27978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "bcf8d79c95544039b2af5aaee160fdca"
format_stamp: "Formatted at 2026-09-28 02:42:42 on dist-test-slave-kkrh"
server_key: "730268f9c96eccbd1c04c783fb970cad"
server_key_iv: "750e041e43d5c279ec57708b72e2df78"
server_key_version: "encryptionkey@0"
I20260928 02:42:42.947335 27978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:42.966554 27978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:42.969152 27978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:42.969331 27978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:42.969589 27978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:42.969923 27978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:42.969981 27978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:42.970026 27978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:42.970057 27978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:42.976418 27978 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:40975
I20260928 02:42:42.976464 28106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:40975 every 8 connection(s)
I20260928 02:42:42.976794 27978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:42:42 dist-test-slave-kkrh krb5kdc[27737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563362, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:42.985437 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 27978
I20260928 02:42:42.985562 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:42:42.985788 9736 external_mini_cluster.cc:1428] Setting key 592842d3e344e697362eeda9d1bd2687
I20260928 02:42:42.985882 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:42.977544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:60741 (local address 127.9.130.62:42155)
0928 02:42:42.977613 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:42.977615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:42.977835 (+ 220us) server_negotiation.cc:407] Connection header received
0928 02:42:42.978371 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:42.978372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:42.978381 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:42.978443 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:42.979414 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:42.979916 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:42.980934 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:42.981107 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:42.981155 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:42.983269 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:42.983270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:42.983271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:42.983277 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:42.984552 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:42.984915 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:42.984916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:42.984916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:42.984962 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:42.985173 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:42.985174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:42.985174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:42.985306 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:42.985377 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:42.985700 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 02:42:42.985789 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:42:42.986126 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:42.977509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:42155 (local address 127.9.130.2:60741)
0928 02:42:42.977696 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:42.977740 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:42.978282 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:42.978527 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:42.978530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:42.979131 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:42.979326 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:42.979331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:42.979996 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:42.979997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:42.980857 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:42.980859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:42.981184 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:42.981185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:42.981291 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:42.981743 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:42.981752 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:42.983182 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:42.984620 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:42.984627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:42.984628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:42.984850 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:42.985024 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:42.985025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:42.985025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:42.985118 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:42.985398 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:42.985404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:42.985516 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 02:42:42.985664 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 02:42:42.985806 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":112,"threads_started":1}
I20260928 02:42:42.986577 28107 heartbeater.cc:347] Connected to a master server at 127.9.130.62:42155
I20260928 02:42:42.986680 28107 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:42.986881 28107 heartbeater.cc:511] Master 127.9.130.62:42155 requested a full tablet report, sending...
I20260928 02:42:42.987277 27782 ts_manager.cc:194] Registered new tserver with Master: bcf8d79c95544039b2af5aaee160fdca (127.9.130.2:40975)
I20260928 02:42:42.987641 27782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:60741
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:42:43.010959 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42155
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:43.090903 28112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:43.091095 28112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:43.091118 28112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:43.092392 28112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:43.092432 28112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:43.092458 28112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:42:43.093879 28112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:42155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:43.094103 28112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:43.094338 28112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:43.099200 28118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:43.099157 28117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:43.099226 28112 server_base.cc:1034] running on GCE node
W20260928 02:42:43.099157 28120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:43.099622 28112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:43.099903 28112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:43.101049 28112 hybrid_clock.cc:648] HybridClock initialized: now 1790563363101038 us; error 26 us; skew 500 ppm
Sep 28 02:42:43 dist-test-slave-kkrh krb5kdc[27737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563363, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:43.103580 28112 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:42:43.104076 28112 webserver.cc:460] Webserver started at http://127.9.130.3:37569/ using document root <none> and password file <none>
I20260928 02:42:43.104295 28112 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:43.104337 28112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:43.104481 28112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:43.105466 28112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "1150f7e221aa47458f9a7126f0303c20"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "fcdc133f72b07d9a1c64f3078d26be1b"
server_key_iv: "0b218c57f5c4be5efba3d80873fec480"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.105809 28112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "1150f7e221aa47458f9a7126f0303c20"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "fcdc133f72b07d9a1c64f3078d26be1b"
server_key_iv: "0b218c57f5c4be5efba3d80873fec480"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.107066 28112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:42:43.107784 28127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.108016 28112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:43.108098 28112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "1150f7e221aa47458f9a7126f0303c20"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "fcdc133f72b07d9a1c64f3078d26be1b"
server_key_iv: "0b218c57f5c4be5efba3d80873fec480"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.108147 28112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:43.121661 28112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:43.123359 28112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:43.123461 28112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:43.123618 28112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:43.123908 28112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:43.123968 28112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.123989 28112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:43.124001 28112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.129487 28112 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:36917
I20260928 02:42:43.129545 28240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:36917 every 8 connection(s)
I20260928 02:42:43.129822 28112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:42:43 dist-test-slave-kkrh krb5kdc[27737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563363, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:43.135583 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 28112
I20260928 02:42:43.135694 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:42:43.135964 9736 external_mini_cluster.cc:1428] Setting key d6f63915589a57b0364ed92da70c9431
I20260928 02:42:43.138759 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.130505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:42183 (local address 127.9.130.62:42155)
0928 02:42:43.130570 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.130572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.130743 (+ 171us) server_negotiation.cc:407] Connection header received
0928 02:42:43.131203 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.131204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.131213 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.131268 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:43.132202 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.132683 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.133596 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.133775 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.133826 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.136195 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:43.136196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:43.136197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:43.136203 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:43.137629 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:43.138039 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:43.138040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:43.138040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:43.138086 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:43.138288 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:43.138289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:43.138289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:43.138382 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:43.138430 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.138644 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.138699 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:42:43.139009 28243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.130464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:42155 (local address 127.9.130.3:42183)
0928 02:42:43.130613 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.130652 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.131117 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.131333 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.131336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.131944 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:43.132124 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.132128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.132757 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.132758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.133521 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.133523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.133844 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.133845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.133939 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.134468 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:43.134475 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:43.136076 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:43.137723 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:43.137730 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:43.137732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:43.137971 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:43.138143 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:43.138144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:43.138144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:43.138235 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:43.138435 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:43.138439 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:43.138541 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.138618 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.138725 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260928 02:42:43.139605 28241 heartbeater.cc:347] Connected to a master server at 127.9.130.62:42155
I20260928 02:42:43.139678 28241 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:43.139880 28241 heartbeater.cc:511] Master 127.9.130.62:42155 requested a full tablet report, sending...
I20260928 02:42:43.140235 27782 ts_manager.cc:194] Registered new tserver with Master: 1150f7e221aa47458f9a7126f0303c20 (127.9.130.3:36917)
I20260928 02:42:43.140559 27782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:42183
I20260928 02:42:43.147722 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:42:43.155488 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.149105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.9.130.62:42155)
0928 02:42:43.149186 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.149188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.149284 (+ 96us) server_negotiation.cc:407] Connection header received
0928 02:42:43.149387 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.149388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.149397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.149494 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:43.150041 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.150520 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.151280 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.151468 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.151568 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.152803 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:43.152804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:43.152804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:43.152812 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:43.153990 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:43.154406 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:43.154407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:43.154408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:43.154512 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:43.154813 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:43.154814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:43.154814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:43.155015 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:43.155144 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.155319 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.155386 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:42:43.157183 27782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:42:43.158699 27782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:42:43.164860 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.160879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.9.130.1:46159)
0928 02:42:43.161251 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.161255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.161264 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:42:43.161309 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.161311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.161328 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.161409 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:43.161539 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.162396 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.163877 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.164472 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.164559 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.164621 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.164736 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.164774 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":126,"threads_started":1}
I20260928 02:42:43.164845 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.160750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46159 (local address 127.0.0.1:42596)
0928 02:42:43.160910 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.160946 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.161010 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.161402 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.161403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.161404 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:43.161441 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.161451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.162498 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.162499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.163764 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.163767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.164558 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.164559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.164636 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.164648 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.164721 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.164772 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
I20260928 02:42:43.165040 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.160984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.9.130.3:36917)
0928 02:42:43.161135 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.161140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.161152 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:42:43.161242 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.161244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.161255 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.161318 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:43.161511 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.162671 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.163927 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.164547 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.164615 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.164664 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.164902 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.164956 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260928 02:42:43.165210 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.160767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:40975 (local address 127.0.0.1:60118)
0928 02:42:43.161538 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.161549 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.161596 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.161862 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.161863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.161865 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:43.161898 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.161899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.163079 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.163080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.164060 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.164063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.164983 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.164984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.165046 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.165050 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.165101 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.165144 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":61,"threads_started":1}
I20260928 02:42:43.165212 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.160766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:36917 (local address 127.0.0.1:59412)
0928 02:42:43.161098 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.161109 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.161163 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.161345 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.161347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.161350 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:43.161422 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.161424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.162773 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.162774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.163833 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.163835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.164698 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.164699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.164761 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.164765 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.164813 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.164857 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":43,"threads_started":1}
I20260928 02:42:43.165553 28254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.160992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.9.130.2:40975)
0928 02:42:43.161170 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.161175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.161658 (+ 483us) server_negotiation.cc:407] Connection header received
0928 02:42:43.161703 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.161705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.161722 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.161787 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:43.161981 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.162985 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.164147 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.164843 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.164901 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.164952 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.165412 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.165468 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":71,"threads_started":1}
I20260928 02:42:43.165665 27907 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4e398339424be88551242888e80d67 (DEFAULT_TABLE table=test-table [id=4abec2ceab07429fa5816333b8750b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:43.166035 27907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4e398339424be88551242888e80d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:43.166092 28041 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4e398339424be88551242888e80d67 (DEFAULT_TABLE table=test-table [id=4abec2ceab07429fa5816333b8750b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:43.166319 28041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4e398339424be88551242888e80d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:43.168707 28255 tablet_bootstrap.cc:492] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820: Bootstrap starting.
I20260928 02:42:43.169168 28256 tablet_bootstrap.cc:492] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca: Bootstrap starting.
I20260928 02:42:43.169121 28175 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4e398339424be88551242888e80d67 (DEFAULT_TABLE table=test-table [id=4abec2ceab07429fa5816333b8750b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:43.169409 28175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4e398339424be88551242888e80d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:43.169430 28255 tablet_bootstrap.cc:654] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:43.169829 28256 tablet_bootstrap.cc:654] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:43.170023 28255 log.cc:824] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:43.170394 28256 log.cc:824] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:43.170773 28255 tablet_bootstrap.cc:492] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820: No bootstrap required, opened a new log
I20260928 02:42:43.170828 28255 ts_tablet_manager.cc:1397] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:43.171231 28256 tablet_bootstrap.cc:492] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca: No bootstrap required, opened a new log
I20260928 02:42:43.171298 28256 ts_tablet_manager.cc:1397] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:43.172376 28259 tablet_bootstrap.cc:492] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20: Bootstrap starting.
I20260928 02:42:43.172832 28255 raft_consensus.cc:348] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ed372cfc6f4fd084bc8efe92186820" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46159 } } peers { permanent_uuid: "bcf8d79c95544039b2af5aaee160fdca" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 40975 } } peers { permanent_uuid: "1150f7e221aa47458f9a7126f0303c20" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36917 } }
I20260928 02:42:43.173017 28255 raft_consensus.cc:374] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:43.173059 28255 raft_consensus.cc:729] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ed372cfc6f4fd084bc8efe92186820, State: Initialized, Role: FOLLOWER
I20260928 02:42:43.173106 28259 tablet_bootstrap.cc:654] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:43.173202 28255 consensus_queue.cc:260] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ed372cfc6f4fd084bc8efe92186820" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46159 } } peers { permanent_uuid: "bcf8d79c95544039b2af5aaee160fdca" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 40975 } } peers { permanent_uuid: "1150f7e221aa47458f9a7126f0303c20" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36917 } }
I20260928 02:42:43.173234 28256 raft_consensus.cc:348] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ed372cfc6f4fd084bc8efe92186820" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46159 } } peers { permanent_uuid: "bcf8d79c95544039b2af5aaee160fdca" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 40975 } } peers { permanent_uuid: "1150f7e221aa47458f9a7126f0303c20" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36917 } }
I20260928 02:42:43.173393 28256 raft_consensus.cc:374] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:43.173434 28256 raft_consensus.cc:729] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf8d79c95544039b2af5aaee160fdca, State: Initialized, Role: FOLLOWER
I20260928 02:42:43.173482 28255 ts_tablet_manager.cc:1428] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:42:43.173548 28256 consensus_queue.cc:260] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ed372cfc6f4fd084bc8efe92186820" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46159 } } peers { permanent_uuid: "bcf8d79c95544039b2af5aaee160fdca" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 40975 } } peers { permanent_uuid: "1150f7e221aa47458f9a7126f0303c20" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36917 } }
I20260928 02:42:43.173592 27973 heartbeater.cc:503] Master 127.9.130.62:42155 was elected leader, sending a full tablet report...
I20260928 02:42:43.173641 28259 log.cc:824] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:43.173795 28256 ts_tablet_manager.cc:1428] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:42:43.173905 28107 heartbeater.cc:503] Master 127.9.130.62:42155 was elected leader, sending a full tablet report...
I20260928 02:42:43.174603 28259 tablet_bootstrap.cc:492] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20: No bootstrap required, opened a new log
I20260928 02:42:43.174664 28259 ts_tablet_manager.cc:1397] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:43.176213 28259 raft_consensus.cc:348] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ed372cfc6f4fd084bc8efe92186820" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46159 } } peers { permanent_uuid: "bcf8d79c95544039b2af5aaee160fdca" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 40975 } } peers { permanent_uuid: "1150f7e221aa47458f9a7126f0303c20" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36917 } }
I20260928 02:42:43.176342 28259 raft_consensus.cc:374] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:43.176362 28259 raft_consensus.cc:729] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1150f7e221aa47458f9a7126f0303c20, State: Initialized, Role: FOLLOWER
I20260928 02:42:43.176465 28259 consensus_queue.cc:260] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ed372cfc6f4fd084bc8efe92186820" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46159 } } peers { permanent_uuid: "bcf8d79c95544039b2af5aaee160fdca" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 40975 } } peers { permanent_uuid: "1150f7e221aa47458f9a7126f0303c20" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36917 } }
I20260928 02:42:43.176700 28259 ts_tablet_manager.cc:1428] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:43.176816 28241 heartbeater.cc:503] Master 127.9.130.62:42155 was elected leader, sending a full tablet report...
I20260928 02:42:43.195894 28261 raft_consensus.cc:482] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:42:43.196031 28261 raft_consensus.cc:504] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ed372cfc6f4fd084bc8efe92186820" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46159 } } peers { permanent_uuid: "bcf8d79c95544039b2af5aaee160fdca" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 40975 } } peers { permanent_uuid: "1150f7e221aa47458f9a7126f0303c20" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36917 } }
I20260928 02:42:43.196416 28261 leader_election.cc:288] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37ed372cfc6f4fd084bc8efe92186820 (127.9.130.1:46159), 1150f7e221aa47458f9a7126f0303c20 (127.9.130.3:36917)
I20260928 02:42:43.199446 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.196616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:36901 (local address 127.9.130.1:46159)
0928 02:42:43.196664 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.196666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.196677 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:42:43.196737 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.196738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.196752 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.196819 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:43.197009 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.197550 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.198592 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.199089 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.199171 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.199181 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.199320 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.199373 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:42:43.199448 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.196561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:36917 (local address 127.9.130.2:60693)
0928 02:42:43.196788 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.196798 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.196852 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.197037 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.197038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.197039 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:43.197102 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.197103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.197752 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.197753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.198608 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.198611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.199231 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.199231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.199293 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.199297 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.199349 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.199377 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":53,"thread_start_us":116,"threads_started":1}
I20260928 02:42:43.199477 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.196660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:60693 (local address 127.9.130.3:36917)
0928 02:42:43.196713 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.196715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.196861 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:42:43.196895 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.196896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.196947 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.197005 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:43.197176 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.197671 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.198690 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.199097 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.199192 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.199202 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.199370 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.199412 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:42:43.199448 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.196556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46159 (local address 127.9.130.2:36901)
0928 02:42:43.196619 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.196631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.196675 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.196847 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.196848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.196851 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:43.196925 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.196927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.197627 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.197628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.198515 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.198517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.199169 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.199171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.199250 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.199253 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.199315 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.199362 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 02:42:43.199801 28195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4e398339424be88551242888e80d67" candidate_uuid: "bcf8d79c95544039b2af5aaee160fdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1150f7e221aa47458f9a7126f0303c20" is_pre_election: true
I20260928 02:42:43.200002 28195 raft_consensus.cc:2454] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf8d79c95544039b2af5aaee160fdca in term 0.
I20260928 02:42:43.200012 27927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4e398339424be88551242888e80d67" candidate_uuid: "bcf8d79c95544039b2af5aaee160fdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ed372cfc6f4fd084bc8efe92186820" is_pre_election: true
I20260928 02:42:43.200138 27927 raft_consensus.cc:2454] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf8d79c95544039b2af5aaee160fdca in term 0.
I20260928 02:42:43.200271 27997 leader_election.cc:302] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1150f7e221aa47458f9a7126f0303c20, bcf8d79c95544039b2af5aaee160fdca; no voters:
I20260928 02:42:43.200390 28261 raft_consensus.cc:2781] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:42:43.200448 28261 raft_consensus.cc:482] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:42:43.200482 28261 raft_consensus.cc:3037] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:43.201131 28261 raft_consensus.cc:504] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ed372cfc6f4fd084bc8efe92186820" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46159 } } peers { permanent_uuid: "bcf8d79c95544039b2af5aaee160fdca" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 40975 } } peers { permanent_uuid: "1150f7e221aa47458f9a7126f0303c20" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36917 } }
I20260928 02:42:43.201274 28261 leader_election.cc:288] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [CANDIDATE]: Term 1 election: Requested vote from peers 37ed372cfc6f4fd084bc8efe92186820 (127.9.130.1:46159), 1150f7e221aa47458f9a7126f0303c20 (127.9.130.3:36917)
I20260928 02:42:43.201421 28195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4e398339424be88551242888e80d67" candidate_uuid: "bcf8d79c95544039b2af5aaee160fdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1150f7e221aa47458f9a7126f0303c20"
I20260928 02:42:43.201432 27927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4e398339424be88551242888e80d67" candidate_uuid: "bcf8d79c95544039b2af5aaee160fdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ed372cfc6f4fd084bc8efe92186820"
I20260928 02:42:43.201494 28195 raft_consensus.cc:3037] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:43.201522 27927 raft_consensus.cc:3037] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:43.202266 27927 raft_consensus.cc:2454] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf8d79c95544039b2af5aaee160fdca in term 1.
I20260928 02:42:43.202301 28195 raft_consensus.cc:2454] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf8d79c95544039b2af5aaee160fdca in term 1.
I20260928 02:42:43.202471 27996 leader_election.cc:302] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ed372cfc6f4fd084bc8efe92186820, bcf8d79c95544039b2af5aaee160fdca; no voters:
I20260928 02:42:43.202572 28261 raft_consensus.cc:2781] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:43.202747 28261 raft_consensus.cc:686] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [term 1 LEADER]: Becoming Leader. State: Replica: bcf8d79c95544039b2af5aaee160fdca, State: Running, Role: LEADER
I20260928 02:42:43.202836 28261 consensus_queue.cc:237] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ed372cfc6f4fd084bc8efe92186820" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46159 } } peers { permanent_uuid: "bcf8d79c95544039b2af5aaee160fdca" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 40975 } } peers { permanent_uuid: "1150f7e221aa47458f9a7126f0303c20" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36917 } }
I20260928 02:42:43.203663 27781 catalog_manager.cc:5495] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf8d79c95544039b2af5aaee160fdca (127.9.130.2). New cstate: current_term: 1 leader_uuid: "bcf8d79c95544039b2af5aaee160fdca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ed372cfc6f4fd084bc8efe92186820" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf8d79c95544039b2af5aaee160fdca" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 40975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1150f7e221aa47458f9a7126f0303c20" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36917 } health_report { overall_health: UNKNOWN } } }
I20260928 02:42:43.216837 28254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.214713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.9.130.2:40975)
0928 02:42:43.214758 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.214759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.214767 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:42:43.214814 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.214814 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.214832 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.214890 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:42:43.214991 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.215442 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.216198 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.216373 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.216435 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.216569 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:42:43.216649 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:42:43.216716 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.216761 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.216768 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:42:43.218546 28195 raft_consensus.cc:1261] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20 [term 1 FOLLOWER]: Refusing update from remote peer bcf8d79c95544039b2af5aaee160fdca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:43.218554 27927 raft_consensus.cc:1261] T 7c4e398339424be88551242888e80d67 P 37ed372cfc6f4fd084bc8efe92186820 [term 1 FOLLOWER]: Refusing update from remote peer bcf8d79c95544039b2af5aaee160fdca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:43.218886 28261 consensus_queue.cc:1035] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1150f7e221aa47458f9a7126f0303c20" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:43.219014 28265 consensus_queue.cc:1035] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "37ed372cfc6f4fd084bc8efe92186820" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 46159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:43.221119 28268 mvcc.cc:205] Tried to move back new op lower bound from 7334147535740870656 to 7334147535679766528. Current Snapshot: MvccSnapshot[applied={T|T < 7334147535740870656}]
I20260928 02:42:43.221570 28270 mvcc.cc:205] Tried to move back new op lower bound from 7334147535740870656 to 7334147535679766528. Current Snapshot: MvccSnapshot[applied={T|T < 7334147535740870656}]
I20260928 02:42:43.221822 28269 mvcc.cc:205] Tried to move back new op lower bound from 7334147535740870656 to 7334147535679766528. Current Snapshot: MvccSnapshot[applied={T|T < 7334147535740870656}]
I20260928 02:42:43.224114 27781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:43.224217 27781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:43.224970 27781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06129bde60234c5eab8cf03aac30c816: Sending DeleteTablet for 3 replicas of tablet 7c4e398339424be88551242888e80d67
I20260928 02:42:43.225277 28041 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c4e398339424be88551242888e80d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:43 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:60118
I20260928 02:42:43.225288 28175 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c4e398339424be88551242888e80d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:43 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:59412
I20260928 02:42:43.225291 27907 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c4e398339424be88551242888e80d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:43 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:42596
I20260928 02:42:43.225603 28279 tablet_replica.cc:330] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca: stopping tablet replica
I20260928 02:42:43.225703 28279 raft_consensus.cc:2229] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:42:43.225777 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 27844
I20260928 02:42:43.226107 28279 raft_consensus.cc:2258] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:43.226451 28278 tablet_replica.cc:330] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20: stopping tablet replica
I20260928 02:42:43.226549 28278 raft_consensus.cc:2229] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:42:43.226656 28278 raft_consensus.cc:2258] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:43.226780 28279 ts_tablet_manager.cc:1905] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:43.227069 28278 ts_tablet_manager.cc:1905] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:43.228616 28279 ts_tablet_manager.cc:1918] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:43.228740 28279 log.cc:1196] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/7c4e398339424be88551242888e80d67
I20260928 02:42:43.229015 28278 ts_tablet_manager.cc:1918] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:43.229104 28278 log.cc:1196] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/7c4e398339424be88551242888e80d67
I20260928 02:42:43.229195 28279 ts_tablet_manager.cc:1939] T 7c4e398339424be88551242888e80d67 P bcf8d79c95544039b2af5aaee160fdca: Deleting consensus metadata
I20260928 02:42:43.229410 28278 ts_tablet_manager.cc:1939] T 7c4e398339424be88551242888e80d67 P 1150f7e221aa47458f9a7126f0303c20: Deleting consensus metadata
I20260928 02:42:43.229643 27766 catalog_manager.cc:4841] TS bcf8d79c95544039b2af5aaee160fdca (127.9.130.2:40975): tablet 7c4e398339424be88551242888e80d67 (table test-table [id=4abec2ceab07429fa5816333b8750b1b]) successfully deleted
I20260928 02:42:43.229769 27769 catalog_manager.cc:4841] TS 1150f7e221aa47458f9a7126f0303c20 (127.9.130.3:36917): tablet 7c4e398339424be88551242888e80d67 (table test-table [id=4abec2ceab07429fa5816333b8750b1b]) successfully deleted
W20260928 02:42:43.232689 27768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:46159 (error 108)
I20260928 02:42:43.233173 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.233039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:46159 (local address 127.0.0.1:42610)
0928 02:42:43.233101 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.233141 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:42:43.233247 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 27978
W20260928 02:42:43.236439 27768 catalog_manager.cc:4584] TS 37ed372cfc6f4fd084bc8efe92186820 (127.9.130.1:46159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c4e398339424be88551242888e80d67: Network error: Client connection negotiation failed: client connection to 127.9.130.1:46159: connect: Connection refused (error 111)
I20260928 02:42:43.238498 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 28112
I20260928 02:42:43.242851 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 27750
2026-09-28T02:42:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1884 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T02:42:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:43Z Disabled control of system clock
I20260928 02:42:43.270114 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:41903
--webserver_interface=127.9.130.62
--webserver_port=0
--builtin_ntp_servers=127.9.130.20:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:41903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:42:43.350525 28285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:43.350749 28285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:43.350771 28285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:43.352051 28285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:43.352092 28285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:43.352103 28285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:43.352116 28285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:43.352128 28285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:43.353464 28285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:41903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:41903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:43.353716 28285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:43.354039 28285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:42:43.359236 28285 server_base.cc:1034] running on GCE node
W20260928 02:42:43.359169 28290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:43.359215 28291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:43.359149 28293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:43.359619 28285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:43.360013 28285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:43.361176 28285 hybrid_clock.cc:648] HybridClock initialized: now 1790563363361162 us; error 29 us; skew 500 ppm
I20260928 02:42:43.362524 28285 webserver.cc:460] Webserver started at http://127.9.130.62:42141/ using document root <none> and password file <none>
I20260928 02:42:43.362778 28285 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:43.362823 28285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:43.362996 28285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:43.364140 28285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "c39d2713dcf8423ebc5487ad06a12882"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "fb840867089018afd7d9fd73a616ae22"
server_key_iv: "3f1e1dc64037d9926a6c37b0bd430bc0"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.364492 28285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "c39d2713dcf8423ebc5487ad06a12882"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "fb840867089018afd7d9fd73a616ae22"
server_key_iv: "3f1e1dc64037d9926a6c37b0bd430bc0"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.365835 28285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:42:43.366629 28299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.366827 28285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:43.366946 28285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "c39d2713dcf8423ebc5487ad06a12882"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "fb840867089018afd7d9fd73a616ae22"
server_key_iv: "3f1e1dc64037d9926a6c37b0bd430bc0"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.367017 28285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:43.376652 28285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:43.376955 28285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:43.377094 28285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:43.381052 28285 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:41903
I20260928 02:42:43.381112 28351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:41903 every 8 connection(s)
I20260928 02:42:43.381448 28285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:43.382431 28352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:43.384224 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 28285
I20260928 02:42:43.384315 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:43.384502 9736 external_mini_cluster.cc:1428] Setting key d1ae224d22ba3285fdf3d7598c3c8408
I20260928 02:42:43.384919 28352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882: Bootstrap starting.
I20260928 02:42:43.385552 28352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:43.385864 28352 log.cc:824] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:43.386570 28352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882: No bootstrap required, opened a new log
I20260928 02:42:43.388352 28352 raft_consensus.cc:348] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39d2713dcf8423ebc5487ad06a12882" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41903 } }
I20260928 02:42:43.388505 28352 raft_consensus.cc:374] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:43.388548 28352 raft_consensus.cc:729] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39d2713dcf8423ebc5487ad06a12882, State: Initialized, Role: FOLLOWER
I20260928 02:42:43.388671 28352 consensus_queue.cc:260] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39d2713dcf8423ebc5487ad06a12882" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41903 } }
I20260928 02:42:43.388752 28352 raft_consensus.cc:388] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:43.388799 28352 raft_consensus.cc:482] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:43.388871 28352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:43.389400 28352 raft_consensus.cc:504] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39d2713dcf8423ebc5487ad06a12882" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41903 } }
I20260928 02:42:43.389513 28352 leader_election.cc:302] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c39d2713dcf8423ebc5487ad06a12882; no voters:
I20260928 02:42:43.389698 28352 leader_election.cc:288] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:43.389814 28357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:43.389979 28357 raft_consensus.cc:686] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [term 1 LEADER]: Becoming Leader. State: Replica: c39d2713dcf8423ebc5487ad06a12882, State: Running, Role: LEADER
I20260928 02:42:43.390010 28352 sys_catalog.cc:564] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:43.390074 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.385054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.9.130.62:41903)
0928 02:42:43.385239 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.385246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.385274 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:42:43.385717 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.385724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.385792 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.385913 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:43.386173 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.387134 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.387971 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.388372 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.388510 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.389258 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:43.389262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:43.389263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:43.389436 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:43.389467 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:43.389478 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:43.389556 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.389764 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.389852 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260928 02:42:43.390069 28357 consensus_queue.cc:237] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39d2713dcf8423ebc5487ad06a12882" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41903 } }
I20260928 02:42:43.390441 28358 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c39d2713dcf8423ebc5487ad06a12882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39d2713dcf8423ebc5487ad06a12882" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41903 } } }
I20260928 02:42:43.390519 28358 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:43.390686 28359 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c39d2713dcf8423ebc5487ad06a12882. Latest consensus state: current_term: 1 leader_uuid: "c39d2713dcf8423ebc5487ad06a12882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39d2713dcf8423ebc5487ad06a12882" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41903 } } }
I20260928 02:42:43.390748 28359 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:43.390946 28365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:43.391397 28365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:43.392796 28365 catalog_manager.cc:1347] Generated new cluster ID: ff8585d0a7054a0abbd60c629885f1aa
I20260928 02:42:43.392848 28365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:43.411772 28365 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:43.412525 28365 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:43.419019 28365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882: Generated new TSK 0
I20260928 02:42:43.419203 28365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:43.422622 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41903
--builtin_ntp_servers=127.9.130.20:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:42:43.502895 28376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:43.503120 28376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:43.503158 28376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:43.504351 28376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:43.504410 28376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:43.504451 28376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:42:43.505803 28376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:43.506047 28376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:43.506333 28376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:43.511228 28382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:43.511228 28384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:43.511397 28376 server_base.cc:1034] running on GCE node
W20260928 02:42:43.511286 28381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:43.511650 28376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:43.511963 28376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:43.513130 28376 hybrid_clock.cc:648] HybridClock initialized: now 1790563363513127 us; error 38 us; skew 500 ppm
I20260928 02:42:43.514266 28376 webserver.cc:460] Webserver started at http://127.9.130.1:36535/ using document root <none> and password file <none>
I20260928 02:42:43.514523 28376 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:43.514591 28376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:43.514739 28376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:43.515796 28376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "200b5eeb2dcb48ba88027f2a94893e5a"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "6c39bfa45cca770088c33d82e2036a38"
server_key_iv: "5139b86f671920d09c14dedbcc52855a"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.516212 28376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "200b5eeb2dcb48ba88027f2a94893e5a"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "6c39bfa45cca770088c33d82e2036a38"
server_key_iv: "5139b86f671920d09c14dedbcc52855a"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.517566 28376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:43.518298 28390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.518451 28376 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:43.518558 28376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "200b5eeb2dcb48ba88027f2a94893e5a"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "6c39bfa45cca770088c33d82e2036a38"
server_key_iv: "5139b86f671920d09c14dedbcc52855a"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.518626 28376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:43.540921 28376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:43.541221 28376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:43.541373 28376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:43.541611 28376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:43.541939 28376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:43.541993 28376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.542035 28376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:43.542062 28376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.547698 28376 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:38993
I20260928 02:42:43.547787 28503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:38993 every 8 connection(s)
I20260928 02:42:43.548139 28376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:42:43.553256 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.548898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:55673 (local address 127.9.130.62:41903)
0928 02:42:43.548962 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.548964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.549189 (+ 225us) server_negotiation.cc:407] Connection header received
0928 02:42:43.549659 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.549660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.549673 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.549731 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:43.550370 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.550863 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.551894 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.552095 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.552151 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.553065 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:43.553066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:43.553067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:43.553070 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:43.553080 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:43.553084 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:43.553124 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.553183 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.553202 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:42:43.553468 28506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.548882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:41903 (local address 127.9.130.1:55673)
0928 02:42:43.549057 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.549098 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.549577 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.549791 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.549794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.549802 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:43.550294 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.550298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.550966 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.550967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.551783 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.551785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.552173 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.552173 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.552295 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.552938 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:43.552938 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:43.552967 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:43.552989 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:43.553124 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:43.553126 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:43.553128 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.553173 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.553251 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260928 02:42:43.553982 28504 heartbeater.cc:347] Connected to a master server at 127.9.130.62:41903
I20260928 02:42:43.554069 28504 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:43.554211 28504 heartbeater.cc:511] Master 127.9.130.62:41903 requested a full tablet report, sending...
I20260928 02:42:43.554840 28316 ts_manager.cc:194] Registered new tserver with Master: 200b5eeb2dcb48ba88027f2a94893e5a (127.9.130.1:38993)
I20260928 02:42:43.555727 28316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.130.1:55673
I20260928 02:42:43.557382 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 28376
I20260928 02:42:43.557480 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:42:43.557617 9736 external_mini_cluster.cc:1428] Setting key 4613958e76e05d2aa2e917a8c8294012
I20260928 02:42:43.558421 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41903
--builtin_ntp_servers=127.9.130.20:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:42:43.637575 28507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:43.637805 28507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:43.637843 28507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:43.638955 28507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:43.639016 28507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:43.639058 28507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:42:43.640389 28507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:43.640641 28507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:43.640926 28507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:43.645838 28512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:43.645864 28515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:43.645895 28513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:43.646175 28507 server_base.cc:1034] running on GCE node
I20260928 02:42:43.646346 28507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:43.646557 28507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:43.647713 28507 hybrid_clock.cc:648] HybridClock initialized: now 1790563363647691 us; error 43 us; skew 500 ppm
I20260928 02:42:43.648965 28507 webserver.cc:460] Webserver started at http://127.9.130.2:44235/ using document root <none> and password file <none>
I20260928 02:42:43.649192 28507 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:43.649235 28507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:43.649396 28507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:43.650421 28507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "baeed387fe354ce993c74418e3cf1dab"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "58d7386cd8fc78bbf6a9582ad9054120"
server_key_iv: "e09353a4dd2706cdd81a7430be248496"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.650763 28507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "baeed387fe354ce993c74418e3cf1dab"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "58d7386cd8fc78bbf6a9582ad9054120"
server_key_iv: "e09353a4dd2706cdd81a7430be248496"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.652066 28507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:42:43.652757 28521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.652944 28507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.653033 28507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "baeed387fe354ce993c74418e3cf1dab"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "58d7386cd8fc78bbf6a9582ad9054120"
server_key_iv: "e09353a4dd2706cdd81a7430be248496"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.653100 28507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:43.674556 28507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:43.674870 28507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:43.675014 28507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:43.675237 28507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:43.675573 28507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:43.675627 28507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.675670 28507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:43.675697 28507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.681402 28507 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:45769
I20260928 02:42:43.681444 28634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:45769 every 8 connection(s)
I20260928 02:42:43.681833 28507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:42:43.682935 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 28507
I20260928 02:42:43.683041 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:42:43.683220 9736 external_mini_cluster.cc:1428] Setting key 72fd1246f2d65291dc837200f32f6b0a
I20260928 02:42:43.684247 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41903
--builtin_ntp_servers=127.9.130.20:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:42:43.688050 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.682583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:48137 (local address 127.9.130.62:41903)
0928 02:42:43.682652 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.682655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.682937 (+ 282us) server_negotiation.cc:407] Connection header received
0928 02:42:43.683423 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.683424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.683436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.683522 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:43.684442 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.684930 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.686447 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.686756 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.686826 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.687707 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:43.687709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:43.687709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:43.687714 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:43.687730 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:43.687735 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:43.687798 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.687910 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.687935 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:42:43.688169 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.682579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:41903 (local address 127.9.130.2:48137)
0928 02:42:43.682815 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.682848 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.683315 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.683617 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.683621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.683630 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:43.684347 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.684352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.685036 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.685037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.686361 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.686364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.686857 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.686858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.686971 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.687543 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:43.687544 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:43.687572 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:43.687594 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:43.687797 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:43.687800 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:43.687802 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.687900 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.687984 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
I20260928 02:42:43.688597 28635 heartbeater.cc:347] Connected to a master server at 127.9.130.62:41903
I20260928 02:42:43.688696 28635 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:43.688896 28635 heartbeater.cc:511] Master 127.9.130.62:41903 requested a full tablet report, sending...
I20260928 02:42:43.689252 28316 ts_manager.cc:194] Registered new tserver with Master: baeed387fe354ce993c74418e3cf1dab (127.9.130.2:45769)
I20260928 02:42:43.689636 28316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.130.2:48137
W20260928 02:42:43.764930 28638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:43.765151 28638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:43.765184 28638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:43.766289 28638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:43.766348 28638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:43.766390 28638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:42:43.767699 28638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:41903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:43.767968 28638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:43.768252 28638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:43.773583 28646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:43.773765 28638 server_base.cc:1034] running on GCE node
W20260928 02:42:43.773581 28644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:43.773626 28643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:43.774111 28638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:43.774351 28638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:43.775516 28638 hybrid_clock.cc:648] HybridClock initialized: now 1790563363775521 us; error 36 us; skew 500 ppm
I20260928 02:42:43.776710 28638 webserver.cc:460] Webserver started at http://127.9.130.3:46639/ using document root <none> and password file <none>
I20260928 02:42:43.776949 28638 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:43.777016 28638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:43.777166 28638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:43.778187 28638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "277baab879bd41c6a1914e32d68814af"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "9dd4a638d0899c268e3cb043ce91d026"
server_key_iv: "5ab4002e06055c2f953db9c7428b6fbd"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.778553 28638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "277baab879bd41c6a1914e32d68814af"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "9dd4a638d0899c268e3cb043ce91d026"
server_key_iv: "5ab4002e06055c2f953db9c7428b6fbd"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.779870 28638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:42:43.780601 28652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.780766 28638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:42:43.780861 28638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "277baab879bd41c6a1914e32d68814af"
format_stamp: "Formatted at 2026-09-28 02:42:43 on dist-test-slave-kkrh"
server_key: "9dd4a638d0899c268e3cb043ce91d026"
server_key_iv: "5ab4002e06055c2f953db9c7428b6fbd"
server_key_version: "encryptionkey@0"
I20260928 02:42:43.780942 28638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:43.796291 28638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:43.796578 28638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:43.796729 28638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:43.796952 28638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:43.797261 28638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:43.797320 28638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.797354 28638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:43.797380 28638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:43.803166 28638 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:34163
I20260928 02:42:43.803213 28765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:34163 every 8 connection(s)
I20260928 02:42:43.803525 28638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:42:43.808665 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.804303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:58145 (local address 127.9.130.62:41903)
0928 02:42:43.804363 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.804365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.804577 (+ 212us) server_negotiation.cc:407] Connection header received
0928 02:42:43.805082 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.805083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.805095 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.805163 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:43.805823 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.806330 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.807278 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.807499 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.807584 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.808400 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:43.808401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:43.808401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:43.808406 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:43.808420 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:43.808424 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:43.808472 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.808563 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.808588 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:42:43.808914 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.804286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:41903 (local address 127.9.130.3:58145)
0928 02:42:43.804463 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.804495 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.804979 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.805234 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.805237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.805245 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:43.805724 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.805728 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.806424 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.806425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.807188 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.807191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.807594 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.807595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.807722 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.808252 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:43.808252 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:43.808290 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:43.808311 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:43.808471 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:43.808476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:43.808479 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.808565 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.808661 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260928 02:42:43.809456 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 28638
I20260928 02:42:43.809553 28766 heartbeater.cc:347] Connected to a master server at 127.9.130.62:41903
I20260928 02:42:43.809602 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:42:43.809664 28766 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:43.809793 9736 external_mini_cluster.cc:1428] Setting key b7fe8c12faa3b60ca4169a69e4bbfa0c
I20260928 02:42:43.809902 28766 heartbeater.cc:511] Master 127.9.130.62:41903 requested a full tablet report, sending...
I20260928 02:42:43.810334 28316 ts_manager.cc:194] Registered new tserver with Master: 277baab879bd41c6a1914e32d68814af (127.9.130.3:34163)
I20260928 02:42:43.810743 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:42:43.810868 28316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.130.3:58145
I20260928 02:42:43.814752 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.812158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.9.130.62:41903)
0928 02:42:43.812201 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.812203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.812262 (+ 59us) server_negotiation.cc:407] Connection header received
0928 02:42:43.812317 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.812317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.812327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.812384 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:43.812511 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.813015 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.813711 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.813917 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.813969 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.814551 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:43.814553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:43.814553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:43.814557 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:43.814569 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:43.814572 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:43.814646 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.814691 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.814701 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:42:43.816365 28316 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:42:43.817996 28316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:42:43.824478 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.820527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.9.130.3:34163)
0928 02:42:43.820813 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.820817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.820824 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:42:43.820850 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.820852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.820863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.820949 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:43.821109 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.822060 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.823328 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.824062 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.824131 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.824209 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.824360 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.824395 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":73,"threads_started":1}
I20260928 02:42:43.824476 28776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.820325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:34163 (local address 127.0.0.1:51552)
0928 02:42:43.820546 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.820557 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.820612 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.820959 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.820962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.820964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:43.821030 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.821032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.822140 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.822141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.823235 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.823238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.824155 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.824157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.824240 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.824253 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.824339 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.824397 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
I20260928 02:42:43.824857 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.820322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:45769 (local address 127.0.0.1:51382)
0928 02:42:43.820473 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.820514 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.820582 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.821046 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.821047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.821049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:43.821083 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.821084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.822608 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.822610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.823670 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.823673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.824614 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.824615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.824683 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.824687 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.824719 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.824785 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":57,"thread_start_us":48,"threads_started":1}
I20260928 02:42:43.825232 28777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.820421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.9.130.2:45769)
0928 02:42:43.820889 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.820893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.820903 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:42:43.820946 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.820948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.820966 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.821048 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:43.821248 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.822488 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.823767 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.824502 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.824971 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.825072 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.825101 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.825141 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":132,"threads_started":1}
I20260928 02:42:43.825348 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.820353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:38993 (local address 127.0.0.1:35186)
0928 02:42:43.820669 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.820677 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.820716 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.821102 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.821103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.821106 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:43.821164 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.821166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.822609 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.822610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.823729 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.823731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.825091 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.825092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.825155 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.825160 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.825212 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.825251 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":38,"thread_start_us":64,"threads_started":1}
I20260928 02:42:43.825232 28700 tablet_service.cc:1467] Processing CreateTablet for tablet 708a298a0b2f4f52aae2417e9cdc5509 (DEFAULT_TABLE table=test-table [id=5b0c29f6f7044c7487b9b8d8c5cc6916]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:43.825595 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.820660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.9.130.1:38993)
0928 02:42:43.820854 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.820859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.820871 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:42:43.820918 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.820920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.820937 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.821027 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:43.821254 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.822525 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.823840 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.824879 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.824969 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.825058 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.825227 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.825263 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260928 02:42:43.825603 28700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708a298a0b2f4f52aae2417e9cdc5509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:43.826139 28438 tablet_service.cc:1467] Processing CreateTablet for tablet 708a298a0b2f4f52aae2417e9cdc5509 (DEFAULT_TABLE table=test-table [id=5b0c29f6f7044c7487b9b8d8c5cc6916]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:43.826426 28438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708a298a0b2f4f52aae2417e9cdc5509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:43.828564 28780 tablet_bootstrap.cc:492] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af: Bootstrap starting.
I20260928 02:42:43.829113 28781 tablet_bootstrap.cc:492] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a: Bootstrap starting.
I20260928 02:42:43.829320 28780 tablet_bootstrap.cc:654] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:43.829236 28569 tablet_service.cc:1467] Processing CreateTablet for tablet 708a298a0b2f4f52aae2417e9cdc5509 (DEFAULT_TABLE table=test-table [id=5b0c29f6f7044c7487b9b8d8c5cc6916]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:43.829587 28569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708a298a0b2f4f52aae2417e9cdc5509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:43.829665 28780 log.cc:824] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:43.829743 28781 tablet_bootstrap.cc:654] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:43.830063 28781 log.cc:824] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:43.830466 28780 tablet_bootstrap.cc:492] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af: No bootstrap required, opened a new log
I20260928 02:42:43.830543 28780 ts_tablet_manager.cc:1397] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:42:43.830756 28781 tablet_bootstrap.cc:492] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a: No bootstrap required, opened a new log
I20260928 02:42:43.830844 28781 ts_tablet_manager.cc:1397] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:42:43.832852 28781 raft_consensus.cc:348] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277baab879bd41c6a1914e32d68814af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34163 } } peers { permanent_uuid: "baeed387fe354ce993c74418e3cf1dab" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45769 } } peers { permanent_uuid: "200b5eeb2dcb48ba88027f2a94893e5a" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38993 } }
I20260928 02:42:43.833001 28781 raft_consensus.cc:374] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:43.833043 28781 raft_consensus.cc:729] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200b5eeb2dcb48ba88027f2a94893e5a, State: Initialized, Role: FOLLOWER
I20260928 02:42:43.833175 28781 consensus_queue.cc:260] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277baab879bd41c6a1914e32d68814af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34163 } } peers { permanent_uuid: "baeed387fe354ce993c74418e3cf1dab" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45769 } } peers { permanent_uuid: "200b5eeb2dcb48ba88027f2a94893e5a" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38993 } }
I20260928 02:42:43.833442 28781 ts_tablet_manager.cc:1428] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:42:43.833534 28504 heartbeater.cc:503] Master 127.9.130.62:41903 was elected leader, sending a full tablet report...
I20260928 02:42:43.833411 28780 raft_consensus.cc:348] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277baab879bd41c6a1914e32d68814af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34163 } } peers { permanent_uuid: "baeed387fe354ce993c74418e3cf1dab" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45769 } } peers { permanent_uuid: "200b5eeb2dcb48ba88027f2a94893e5a" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38993 } }
I20260928 02:42:43.833590 28780 raft_consensus.cc:374] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:43.833636 28780 raft_consensus.cc:729] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277baab879bd41c6a1914e32d68814af, State: Initialized, Role: FOLLOWER
I20260928 02:42:43.833763 28780 consensus_queue.cc:260] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277baab879bd41c6a1914e32d68814af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34163 } } peers { permanent_uuid: "baeed387fe354ce993c74418e3cf1dab" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45769 } } peers { permanent_uuid: "200b5eeb2dcb48ba88027f2a94893e5a" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38993 } }
I20260928 02:42:43.834029 28780 ts_tablet_manager.cc:1428] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 02:42:43.834093 28766 heartbeater.cc:503] Master 127.9.130.62:41903 was elected leader, sending a full tablet report...
I20260928 02:42:43.834262 28784 tablet_bootstrap.cc:492] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab: Bootstrap starting.
I20260928 02:42:43.834975 28784 tablet_bootstrap.cc:654] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:43.835323 28784 log.cc:824] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:43.836054 28784 tablet_bootstrap.cc:492] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab: No bootstrap required, opened a new log
I20260928 02:42:43.836139 28784 ts_tablet_manager.cc:1397] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:42:43.838053 28784 raft_consensus.cc:348] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277baab879bd41c6a1914e32d68814af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34163 } } peers { permanent_uuid: "baeed387fe354ce993c74418e3cf1dab" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45769 } } peers { permanent_uuid: "200b5eeb2dcb48ba88027f2a94893e5a" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38993 } }
I20260928 02:42:43.838209 28784 raft_consensus.cc:374] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:43.838258 28784 raft_consensus.cc:729] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baeed387fe354ce993c74418e3cf1dab, State: Initialized, Role: FOLLOWER
I20260928 02:42:43.838407 28784 consensus_queue.cc:260] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277baab879bd41c6a1914e32d68814af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34163 } } peers { permanent_uuid: "baeed387fe354ce993c74418e3cf1dab" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45769 } } peers { permanent_uuid: "200b5eeb2dcb48ba88027f2a94893e5a" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38993 } }
I20260928 02:42:43.838665 28784 ts_tablet_manager.cc:1428] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:42:43.838728 28635 heartbeater.cc:503] Master 127.9.130.62:41903 was elected leader, sending a full tablet report...
I20260928 02:42:43.845736 28788 raft_consensus.cc:482] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:42:43.845880 28788 raft_consensus.cc:504] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277baab879bd41c6a1914e32d68814af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34163 } } peers { permanent_uuid: "baeed387fe354ce993c74418e3cf1dab" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45769 } } peers { permanent_uuid: "200b5eeb2dcb48ba88027f2a94893e5a" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38993 } }
I20260928 02:42:43.846230 28788 leader_election.cc:288] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 277baab879bd41c6a1914e32d68814af (127.9.130.3:34163), 200b5eeb2dcb48ba88027f2a94893e5a (127.9.130.1:38993)
I20260928 02:42:43.849972 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.846475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:38525 (local address 127.9.130.3:34163)
0928 02:42:43.846542 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.846544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.846554 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:42:43.846582 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.846583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.846591 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.846670 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:43.846834 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.847572 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.849062 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.849608 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.849685 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.849694 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.849867 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.849909 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:42:43.849980 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.846454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:47823 (local address 127.9.130.1:38993)
0928 02:42:43.846502 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.846504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.846625 (+ 121us) server_negotiation.cc:407] Connection header received
0928 02:42:43.846649 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.846650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.846661 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.846708 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:43.846838 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.847583 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.849063 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.849609 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.849685 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.849694 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.849881 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.849923 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:42:43.849980 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.846358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:34163 (local address 127.9.130.2:38525)
0928 02:42:43.846409 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.846421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.846467 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.846684 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.846684 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.846686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:43.846755 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.846756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.847670 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.847671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.848968 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.848971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.849692 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.849693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.849774 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.849779 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.849838 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.849885 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":104}
I20260928 02:42:43.849993 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.846403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:38993 (local address 127.9.130.2:47823)
0928 02:42:43.846559 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.846566 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:43.846609 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:43.846717 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:43.846718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:43.846719 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:43.846776 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.846777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.847670 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.847671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.848977 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:43.848979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.849695 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.849695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:43.849810 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.849814 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:42:43.849866 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:42:43.849892 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":68,"thread_start_us":87,"threads_started":1}
I20260928 02:42:43.850297 28458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708a298a0b2f4f52aae2417e9cdc5509" candidate_uuid: "baeed387fe354ce993c74418e3cf1dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "200b5eeb2dcb48ba88027f2a94893e5a" is_pre_election: true
I20260928 02:42:43.850318 28720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708a298a0b2f4f52aae2417e9cdc5509" candidate_uuid: "baeed387fe354ce993c74418e3cf1dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277baab879bd41c6a1914e32d68814af" is_pre_election: true
I20260928 02:42:43.850430 28720 raft_consensus.cc:2454] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baeed387fe354ce993c74418e3cf1dab in term 0.
I20260928 02:42:43.850430 28458 raft_consensus.cc:2454] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baeed387fe354ce993c74418e3cf1dab in term 0.
I20260928 02:42:43.850634 28525 leader_election.cc:302] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 200b5eeb2dcb48ba88027f2a94893e5a, baeed387fe354ce993c74418e3cf1dab; no voters:
I20260928 02:42:43.850749 28788 raft_consensus.cc:2781] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:42:43.850832 28788 raft_consensus.cc:482] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:42:43.850865 28788 raft_consensus.cc:3037] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:43.851493 28788 raft_consensus.cc:504] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277baab879bd41c6a1914e32d68814af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34163 } } peers { permanent_uuid: "baeed387fe354ce993c74418e3cf1dab" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45769 } } peers { permanent_uuid: "200b5eeb2dcb48ba88027f2a94893e5a" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38993 } }
I20260928 02:42:43.851624 28788 leader_election.cc:288] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [CANDIDATE]: Term 1 election: Requested vote from peers 277baab879bd41c6a1914e32d68814af (127.9.130.3:34163), 200b5eeb2dcb48ba88027f2a94893e5a (127.9.130.1:38993)
I20260928 02:42:43.851747 28720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708a298a0b2f4f52aae2417e9cdc5509" candidate_uuid: "baeed387fe354ce993c74418e3cf1dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277baab879bd41c6a1914e32d68814af"
I20260928 02:42:43.851789 28458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708a298a0b2f4f52aae2417e9cdc5509" candidate_uuid: "baeed387fe354ce993c74418e3cf1dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "200b5eeb2dcb48ba88027f2a94893e5a"
I20260928 02:42:43.851853 28720 raft_consensus.cc:3037] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:43.851897 28458 raft_consensus.cc:3037] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:43.852735 28720 raft_consensus.cc:2454] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baeed387fe354ce993c74418e3cf1dab in term 1.
I20260928 02:42:43.852735 28458 raft_consensus.cc:2454] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baeed387fe354ce993c74418e3cf1dab in term 1.
I20260928 02:42:43.852918 28524 leader_election.cc:302] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277baab879bd41c6a1914e32d68814af, baeed387fe354ce993c74418e3cf1dab; no voters:
I20260928 02:42:43.853022 28788 raft_consensus.cc:2781] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:43.853171 28788 raft_consensus.cc:686] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [term 1 LEADER]: Becoming Leader. State: Replica: baeed387fe354ce993c74418e3cf1dab, State: Running, Role: LEADER
I20260928 02:42:43.853279 28788 consensus_queue.cc:237] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277baab879bd41c6a1914e32d68814af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34163 } } peers { permanent_uuid: "baeed387fe354ce993c74418e3cf1dab" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45769 } } peers { permanent_uuid: "200b5eeb2dcb48ba88027f2a94893e5a" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38993 } }
I20260928 02:42:43.853988 28316 catalog_manager.cc:5495] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab reported cstate change: term changed from 0 to 1, leader changed from <none> to baeed387fe354ce993c74418e3cf1dab (127.9.130.2). New cstate: current_term: 1 leader_uuid: "baeed387fe354ce993c74418e3cf1dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277baab879bd41c6a1914e32d68814af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baeed387fe354ce993c74418e3cf1dab" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "200b5eeb2dcb48ba88027f2a94893e5a" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38993 } health_report { overall_health: UNKNOWN } } }
I20260928 02:42:43.859365 28777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.857360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.9.130.2:45769)
0928 02:42:43.857395 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:43.857396 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:43.857424 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:42:43.857475 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:43.857476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:43.857489 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:43.857551 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:42:43.857639 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.858062 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.858747 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:43.858920 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:43.858970 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:43.859096 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:42:43.859171 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:42:43.859235 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:43.859292 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 02:42:43.859304 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:42:43.860939 28720 raft_consensus.cc:1261] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af [term 1 FOLLOWER]: Refusing update from remote peer baeed387fe354ce993c74418e3cf1dab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:43.860960 28458 raft_consensus.cc:1261] T 708a298a0b2f4f52aae2417e9cdc5509 P 200b5eeb2dcb48ba88027f2a94893e5a [term 1 FOLLOWER]: Refusing update from remote peer baeed387fe354ce993c74418e3cf1dab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:43.861308 28788 consensus_queue.cc:1035] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "200b5eeb2dcb48ba88027f2a94893e5a" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 38993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:43.861426 28790 consensus_queue.cc:1035] T 708a298a0b2f4f52aae2417e9cdc5509 P baeed387fe354ce993c74418e3cf1dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "277baab879bd41c6a1914e32d68814af" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:43.863746 28793 mvcc.cc:205] Tried to move back new op lower bound from 7334147538372374528 to 7334147538343895040. Current Snapshot: MvccSnapshot[applied={T|T < 7334147538372374528}]
I20260928 02:42:43.864298 28795 mvcc.cc:205] Tried to move back new op lower bound from 7334147538372374528 to 7334147538343895040. Current Snapshot: MvccSnapshot[applied={T|T < 7334147538372374528}]
I20260928 02:42:43.864657 28794 mvcc.cc:205] Tried to move back new op lower bound from 7334147538372374528 to 7334147538343895040. Current Snapshot: MvccSnapshot[applied={T|T < 7334147538372374528 or (T in {7334147538372374528})}]
I20260928 02:42:43.866211 28316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:43.866289 28316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:43.867014 28316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c39d2713dcf8423ebc5487ad06a12882: Sending DeleteTablet for 3 replicas of tablet 708a298a0b2f4f52aae2417e9cdc5509
I20260928 02:42:43.867324 28700 tablet_service.cc:1514] Processing DeleteTablet for tablet 708a298a0b2f4f52aae2417e9cdc5509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:43 UTC) from {username='slave'} at 127.0.0.1:51552
I20260928 02:42:43.867313 28569 tablet_service.cc:1514] Processing DeleteTablet for tablet 708a298a0b2f4f52aae2417e9cdc5509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:43 UTC) from {username='slave'} at 127.0.0.1:51382
I20260928 02:42:43.867341 28438 tablet_service.cc:1514] Processing DeleteTablet for tablet 708a298a0b2f4f52aae2417e9cdc5509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:43 UTC) from {username='slave'} at 127.0.0.1:35186
I20260928 02:42:43.867559 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 28376
I20260928 02:42:43.868592 28804 tablet_replica.cc:330] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af: stopping tablet replica
I20260928 02:42:43.868732 28804 raft_consensus.cc:2229] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:42:43.868844 28804 raft_consensus.cc:2258] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:43.869309 28804 ts_tablet_manager.cc:1905] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:43.871277 28804 ts_tablet_manager.cc:1918] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:43.871351 28804 log.cc:1196] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/708a298a0b2f4f52aae2417e9cdc5509
I20260928 02:42:43.871585 28804 ts_tablet_manager.cc:1939] T 708a298a0b2f4f52aae2417e9cdc5509 P 277baab879bd41c6a1914e32d68814af: Deleting consensus metadata
I20260928 02:42:43.872051 28302 catalog_manager.cc:4841] TS 277baab879bd41c6a1914e32d68814af (127.9.130.3:34163): tablet 708a298a0b2f4f52aae2417e9cdc5509 (table test-table [id=5b0c29f6f7044c7487b9b8d8c5cc6916]) successfully deleted
W20260928 02:42:43.873773 28303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:38993 (error 108)
I20260928 02:42:43.874256 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.874110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:38993 (local address 127.0.0.1:35196)
0928 02:42:43.874174 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.874222 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 02:42:43.874327 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 28507
W20260928 02:42:43.875209 28303 catalog_manager.cc:4584] TS 200b5eeb2dcb48ba88027f2a94893e5a (127.9.130.1:38993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 708a298a0b2f4f52aae2417e9cdc5509: Network error: Client connection negotiation failed: client connection to 127.9.130.1:38993: connect: Connection refused (error 111)
I20260928 02:42:43.879209 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 28638
I20260928 02:42:43.879233 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:43.879141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:45769 (local address 127.0.0.1:51404)
0928 02:42:43.879179 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:43.879210 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.2:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 02:42:43.883991 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 28285
2026-09-28T02:42:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (640 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:43 dist-test-slave-kkrh krb5kdc[28809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:43 dist-test-slave-kkrh krb5kdc[28809](info): set up 2 sockets
Sep 28 02:42:43 dist-test-slave-kkrh krb5kdc[28809](info): commencing operation
krb5kdc: starting...
W20260928 02:42:44.919605 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:42:44 dist-test-slave-kkrh krb5kdc[28809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:42:45.038519 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:37159
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:37159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:45.117555 28822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:45.117784 28822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:45.117817 28822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:45.118957 28822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:45.119035 28822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:45.119063 28822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:45.119094 28822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:45.119118 28822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:45.120628 28822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:37159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:37159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:45.120898 28822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:45.121127 28822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:45.126000 28828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:45.126012 28827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:45.126080 28822 server_base.cc:1034] running on GCE node
W20260928 02:42:45.126019 28830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:45.126432 28822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:45.126724 28822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:45.127880 28822 hybrid_clock.cc:648] HybridClock initialized: now 1790563365127857 us; error 46 us; skew 500 ppm
Sep 28 02:42:45 dist-test-slave-kkrh krb5kdc[28809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563365, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:45.130255 28822 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:42:45.130713 28822 webserver.cc:460] Webserver started at http://127.9.130.62:40113/ using document root <none> and password file <none>
I20260928 02:42:45.130901 28822 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:45.130940 28822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:45.131026 28822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:45.132061 28822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "e2ebb303ab254d20b88474026b6cfd35"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "d5126b1b7233d857ee42d5666ab3e008"
server_key_iv: "1fe84232c5f8eee2be2d61dcbd48b9f0"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.132354 28822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "e2ebb303ab254d20b88474026b6cfd35"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "d5126b1b7233d857ee42d5666ab3e008"
server_key_iv: "1fe84232c5f8eee2be2d61dcbd48b9f0"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.133585 28822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:45.134246 28837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:45.134433 28822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:45.134523 28822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "e2ebb303ab254d20b88474026b6cfd35"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "d5126b1b7233d857ee42d5666ab3e008"
server_key_iv: "1fe84232c5f8eee2be2d61dcbd48b9f0"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.134577 28822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:45.146874 28822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:45.149088 28822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:45.149196 28822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:45.152930 28822 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:37159
I20260928 02:42:45.152990 28889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:37159 every 8 connection(s)
I20260928 02:42:45.153311 28822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:45.154186 28890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:45.156500 28890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35: Bootstrap starting.
I20260928 02:42:45.157065 28890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:45.157342 28890 log.cc:824] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:45.158005 28890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35: No bootstrap required, opened a new log
I20260928 02:42:45.159556 28890 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ebb303ab254d20b88474026b6cfd35" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 37159 } }
I20260928 02:42:45.159713 28890 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:45.159760 28890 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ebb303ab254d20b88474026b6cfd35, State: Initialized, Role: FOLLOWER
I20260928 02:42:45.159945 28890 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ebb303ab254d20b88474026b6cfd35" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 37159 } }
I20260928 02:42:45.160055 28890 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:45.160100 28890 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:45.160149 28890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:45.160655 28890 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ebb303ab254d20b88474026b6cfd35" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 37159 } }
I20260928 02:42:45.160770 28890 leader_election.cc:302] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2ebb303ab254d20b88474026b6cfd35; no voters:
I20260928 02:42:45.160945 28890 leader_election.cc:288] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:45.160983 28893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:45.161175 28893 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [term 1 LEADER]: Becoming Leader. State: Replica: e2ebb303ab254d20b88474026b6cfd35, State: Running, Role: LEADER
I20260928 02:42:45.161229 28890 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:45.161259 28893 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ebb303ab254d20b88474026b6cfd35" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 37159 } }
I20260928 02:42:45.161696 28895 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2ebb303ab254d20b88474026b6cfd35. Latest consensus state: current_term: 1 leader_uuid: "e2ebb303ab254d20b88474026b6cfd35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ebb303ab254d20b88474026b6cfd35" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 37159 } } }
I20260928 02:42:45.161779 28895 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:45.161943 28894 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2ebb303ab254d20b88474026b6cfd35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ebb303ab254d20b88474026b6cfd35" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 37159 } } }
I20260928 02:42:45.162093 28894 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:45.162217 28902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:45.162744 28902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:45.163167 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 28822
I20260928 02:42:45.163285 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:45.163492 9736 external_mini_cluster.cc:1428] Setting key ff3841315819f27dc468ff4c4099ca22
I20260928 02:42:45.164994 28902 catalog_manager.cc:1347] Generated new cluster ID: 55fa82775e5749089db7cd3b22bdcaa2
I20260928 02:42:45.165046 28902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:42:45 dist-test-slave-kkrh krb5kdc[28809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:45.173003 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.163936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35114 (local address 127.9.130.62:37159)
0928 02:42:45.164212 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:45.164221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:45.164256 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:42:45.164368 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:45.164375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:45.164427 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:45.164587 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:45.165276 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.166198 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.166944 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.167361 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.167482 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.169770 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:45.169774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:45.169774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:45.169786 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:45.171397 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:45.171862 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:45.171865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:45.171865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:45.171972 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:45.172189 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:45.172190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:45.172191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:45.172425 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:45.172605 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:45.172677 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 02:42:45.172824 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
I20260928 02:42:45.191577 28902 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:45.192186 28902 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:45.199579 28902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35: Generated new TSK 0
I20260928 02:42:45.199776 28902 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.130.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
Entry for principal kudu/127.9.130.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab.
I20260928 02:42:45.230929 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:37159
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:45.328066 28916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:45.328251 28916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:45.328274 28916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:45.329372 28916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:45.329409 28916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:45.329434 28916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:42:45.330773 28916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.1.keytab
--principal=kudu/127.9.130.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:37159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:45.330979 28916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:45.331172 28916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:45.336076 28921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:45.336086 28924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:45.336280 28916 server_base.cc:1034] running on GCE node
W20260928 02:42:45.336086 28922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:45.336551 28916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:45.336786 28916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:45.337939 28916 hybrid_clock.cc:648] HybridClock initialized: now 1790563365337939 us; error 38 us; skew 500 ppm
Sep 28 02:42:45 dist-test-slave-kkrh krb5kdc[28809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563365, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:45.340415 28916 init.cc:377] Logged in from keytab as kudu/127.9.130.1@KRBTEST.COM (short username kudu)
I20260928 02:42:45.340888 28916 webserver.cc:460] Webserver started at http://127.9.130.1:41603/ using document root <none> and password file <none>
I20260928 02:42:45.341117 28916 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:45.341185 28916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:45.341333 28916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:45.342374 28916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "f123b460a85d4757a2bc3ea72d29813b"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "a2a1ec17a1348f0cab10b2d366735f00"
server_key_iv: "f5dbade54b50e67a9b4a1867a1c19f7f"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.342722 28916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "f123b460a85d4757a2bc3ea72d29813b"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "a2a1ec17a1348f0cab10b2d366735f00"
server_key_iv: "f5dbade54b50e67a9b4a1867a1c19f7f"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.344080 28916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:42:45.344805 28931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:45.344982 28916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:45.345075 28916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "f123b460a85d4757a2bc3ea72d29813b"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "a2a1ec17a1348f0cab10b2d366735f00"
server_key_iv: "f5dbade54b50e67a9b4a1867a1c19f7f"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.345151 28916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:45.364329 28916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:45.366199 28916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:45.366350 28916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:45.366581 28916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:45.366873 28916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:45.366925 28916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:45.366961 28916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:45.366989 28916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:45.373477 28916 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:36779
I20260928 02:42:45.373518 29044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:36779 every 8 connection(s)
I20260928 02:42:45.373831 28916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:42:45.376259 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 28916
I20260928 02:42:45.376390 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:42:45.376582 9736 external_mini_cluster.cc:1428] Setting key 888bc63d8b1ea526813a98f94c59752a
Sep 28 02:42:45 dist-test-slave-kkrh krb5kdc[28809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563365, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.1@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:45.383550 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.374530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:43885 (local address 127.9.130.62:37159)
0928 02:42:45.374593 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:45.374595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:45.374777 (+ 182us) server_negotiation.cc:407] Connection header received
0928 02:42:45.375231 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:45.375232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:45.375245 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:45.375298 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:45.376284 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.376942 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.377990 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.378331 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.378420 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.380744 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:45.380745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:45.380745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:45.380755 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:45.382193 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:45.382602 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:45.382604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:45.382604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:45.382664 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:45.382922 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:45.382923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:45.382924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:45.383034 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:45.383108 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:45.383376 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 02:42:45.383462 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:42:45.383720 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.374504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:37159 (local address 127.9.130.1:43885)
0928 02:42:45.374642 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:45.374682 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:45.375144 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:45.375357 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:45.375360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:45.375927 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:45.376176 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.376185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.377025 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.377026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.377901 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.377903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.378438 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.378439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.378597 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.379094 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:45.379105 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:45.380627 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:45.382293 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:45.382300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:45.382302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:45.382524 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:45.382731 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:45.382732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:45.382732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:45.382840 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:45.383099 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:45.383106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:45.383242 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 02:42:45.383348 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:42:45.383496 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260928 02:42:45.384243 29045 heartbeater.cc:347] Connected to a master server at 127.9.130.62:37159
I20260928 02:42:45.384346 29045 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:45.384521 29045 heartbeater.cc:511] Master 127.9.130.62:37159 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.130.2@KRBTEST.COM; defaulting to no policy
I20260928 02:42:45.385032 28854 ts_manager.cc:194] Registered new tserver with Master: f123b460a85d4757a2bc3ea72d29813b (127.9.130.1:36779)
I20260928 02:42:45.385617 28854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.1@KRBTEST.COM'} at 127.9.130.1:43885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
Entry for principal kudu/127.9.130.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab.
I20260928 02:42:45.400986 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:37159
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:45.481421 29050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:45.481624 29050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:45.481647 29050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:45.482702 29050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:45.482734 29050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:45.482760 29050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:42:45.484171 29050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.2.keytab
--principal=kudu/127.9.130.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:37159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:45.484359 29050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:45.484532 29050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:45.489472 29055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:45.489488 29056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:45.489638 29050 server_base.cc:1034] running on GCE node
W20260928 02:42:45.489523 29058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:45.489931 29050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:45.490180 29050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:45.491340 29050 hybrid_clock.cc:648] HybridClock initialized: now 1790563365491334 us; error 37 us; skew 500 ppm
Sep 28 02:42:45 dist-test-slave-kkrh krb5kdc[28809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563365, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:45.493788 29050 init.cc:377] Logged in from keytab as kudu/127.9.130.2@KRBTEST.COM (short username kudu)
I20260928 02:42:45.494268 29050 webserver.cc:460] Webserver started at http://127.9.130.2:33659/ using document root <none> and password file <none>
I20260928 02:42:45.494503 29050 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:45.494571 29050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:45.494730 29050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:45.495798 29050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "6ee7544de0464e26a93d4bb3555ea90e"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "7e7a9dbded3995b90e747126bcb2651f"
server_key_iv: "c9240f3c53ac9f71534fb9cde2960533"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.496236 29050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "6ee7544de0464e26a93d4bb3555ea90e"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "7e7a9dbded3995b90e747126bcb2651f"
server_key_iv: "c9240f3c53ac9f71534fb9cde2960533"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.498221 29050 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:42:45.498961 29065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:45.499151 29050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:42:45.499253 29050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "6ee7544de0464e26a93d4bb3555ea90e"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "7e7a9dbded3995b90e747126bcb2651f"
server_key_iv: "c9240f3c53ac9f71534fb9cde2960533"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.499339 29050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:45.512678 29050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:45.514688 29050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:45.514861 29050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:45.515100 29050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:45.515419 29050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:45.515475 29050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:45.515514 29050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:45.515540 29050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:45.521349 29050 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:33873
I20260928 02:42:45.521397 29178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:33873 every 8 connection(s)
I20260928 02:42:45.521706 29050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:42:45.525512 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 29050
I20260928 02:42:45.525643 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:42:45.525846 9736 external_mini_cluster.cc:1428] Setting key 5450b797c713bf93245e5b0c96984f35
Sep 28 02:42:45 dist-test-slave-kkrh krb5kdc[28809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563365, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.2@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:45.532661 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.522526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:34681 (local address 127.9.130.62:37159)
0928 02:42:45.522578 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:45.522581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:45.522849 (+ 268us) server_negotiation.cc:407] Connection header received
0928 02:42:45.523343 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:45.523344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:45.523356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:45.523423 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:45.524433 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.524934 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.525976 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.526252 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.526318 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.528807 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:45.528810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:45.528810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:45.528818 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:45.530859 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:45.531448 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:45.531449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:45.531450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:45.531514 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:45.531847 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:45.531848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:45.531849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:45.532011 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:45.532062 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:45.532457 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 02:42:45.532554 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:42:45.533213 29181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.522551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:37159 (local address 127.9.130.2:34681)
0928 02:42:45.522724 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:45.522759 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:45.523221 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:45.523499 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:45.523502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:45.524138 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:45.524325 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.524330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.525016 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.525016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.525885 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.525888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.526339 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.526340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.526481 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.527066 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:45.527079 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:45.528722 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:45.530940 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:45.530946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:45.530948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:45.531349 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:45.531598 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:45.531598 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:45.531599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:45.531739 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:45.532117 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:45.532124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:45.532294 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 02:42:45.532765 (+ 471us) client_negotiation.cc:240] Negotiation successful
0928 02:42:45.532961 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260928 02:42:45.534463 29179 heartbeater.cc:347] Connected to a master server at 127.9.130.62:37159
I20260928 02:42:45.534578 29179 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:45.534781 29179 heartbeater.cc:511] Master 127.9.130.62:37159 requested a full tablet report, sending...
I20260928 02:42:45.535230 28854 ts_manager.cc:194] Registered new tserver with Master: 6ee7544de0464e26a93d4bb3555ea90e (127.9.130.2:33873)
I20260928 02:42:45.535730 28854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.2@KRBTEST.COM'} at 127.9.130.2:34681
WARNING: no policy specified for kudu/127.9.130.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
Entry for principal kudu/127.9.130.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab.
I20260928 02:42:45.554059 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:37159
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:36821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:45.633466 29184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:45.633705 29184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:45.633744 29184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:45.634927 29184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:45.635005 29184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:45.635051 29184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:42:45.636528 29184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:36821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.3.keytab
--principal=kudu/127.9.130.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:37159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:45.636792 29184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:45.637076 29184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:45.642251 29190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:45.642243 29192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:45.642354 29184 server_base.cc:1034] running on GCE node
W20260928 02:42:45.642243 29189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:45.642704 29184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:45.642892 29184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:45.644023 29184 hybrid_clock.cc:648] HybridClock initialized: now 1790563365644011 us; error 28 us; skew 500 ppm
Sep 28 02:42:45 dist-test-slave-kkrh krb5kdc[28809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563365, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:45.646412 29184 init.cc:377] Logged in from keytab as kudu/127.9.130.3@KRBTEST.COM (short username kudu)
I20260928 02:42:45.646883 29184 webserver.cc:460] Webserver started at http://127.9.130.3:43433/ using document root <none> and password file <none>
I20260928 02:42:45.647110 29184 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:45.647156 29184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:45.647297 29184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:45.648440 29184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "ee2fecb0a8064ed493eeded534f39bd6"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "95799fcae85463c6573087e50c03fd3f"
server_key_iv: "0ccaad857d0bb6067b5ade0f120b43e6"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.648800 29184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "ee2fecb0a8064ed493eeded534f39bd6"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "95799fcae85463c6573087e50c03fd3f"
server_key_iv: "0ccaad857d0bb6067b5ade0f120b43e6"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.650086 29184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:42:45.650828 29199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:45.650988 29184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:45.651082 29184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "ee2fecb0a8064ed493eeded534f39bd6"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "95799fcae85463c6573087e50c03fd3f"
server_key_iv: "0ccaad857d0bb6067b5ade0f120b43e6"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.651163 29184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:45.665823 29184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:45.667582 29184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:45.667737 29184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:45.667974 29184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:45.668279 29184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:45.668336 29184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:45.668378 29184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:45.668406 29184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:45.674341 29184 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:36085
I20260928 02:42:45.674387 29312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:36085 every 8 connection(s)
I20260928 02:42:45.674724 29184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:42:45.678597 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 29184
I20260928 02:42:45.678689 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:42:45.678894 9736 external_mini_cluster.cc:1428] Setting key bf53b5e0c27e49ec7d1aadcf2629d715
Sep 28 02:42:45 dist-test-slave-kkrh krb5kdc[28809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563365, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.3@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:42:45.683369 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.675350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:37055 (local address 127.9.130.62:37159)
0928 02:42:45.675422 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:45.675424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:45.675621 (+ 197us) server_negotiation.cc:407] Connection header received
0928 02:42:45.676155 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:45.676156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:45.676166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:45.676222 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:45.677095 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.677578 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.678540 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.678780 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.678834 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.681094 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:45.681095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:45.681096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:45.681102 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:45.682277 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:45.682632 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:45.682633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:45.682633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:45.682680 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:45.682889 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:45.682890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:45.682890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:45.682985 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:45.683034 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:45.683250 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 02:42:45.683306 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:42:45.683574 29315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.675346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:37159 (local address 127.9.130.3:37055)
0928 02:42:45.675493 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:45.675534 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:45.676070 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:45.676272 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:45.676276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:45.676836 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:45.677020 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.677023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.677653 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.677653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.678460 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.678462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.678859 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.678861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.679004 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.679573 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:45.679585 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:45.681011 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:45.682345 (+ 1334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:45.682350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:45.682351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:45.682568 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:45.682737 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:45.682737 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:45.682738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:45.682830 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:45.683036 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:45.683039 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:45.683146 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 02:42:45.683223 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:42:45.683334 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260928 02:42:45.684051 29313 heartbeater.cc:347] Connected to a master server at 127.9.130.62:37159
I20260928 02:42:45.684141 29313 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:45.684283 29313 heartbeater.cc:511] Master 127.9.130.62:37159 requested a full tablet report, sending...
I20260928 02:42:45.684634 28854 ts_manager.cc:194] Registered new tserver with Master: ee2fecb0a8064ed493eeded534f39bd6 (127.9.130.3:36085)
I20260928 02:42:45.684998 28854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.130.3@KRBTEST.COM'} at 127.9.130.3:37055
I20260928 02:42:45.690510 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:42:45.697132 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.691722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.9.130.62:37159)
0928 02:42:45.691766 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:45.691768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:45.691880 (+ 112us) server_negotiation.cc:407] Connection header received
0928 02:42:45.691952 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:45.691953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:45.691960 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:45.692008 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:45.692503 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.692944 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.693603 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.693769 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.693815 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.694950 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:45.694951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:45.694952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:45.694956 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:45.696155 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:45.696475 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:45.696476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:45.696476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:45.696532 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:45.696706 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:45.696707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:45.696707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:45.696838 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:45.696912 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:45.697014 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:42:45.697073 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:42:45.698588 28854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:42:45.700400 28854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:42:45.706588 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.702706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:36779 (local address 127.0.0.1:42034)
0928 02:42:45.702969 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:45.702978 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:45.703021 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:45.703391 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:45.703391 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:45.703393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:45.703424 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.703425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.704780 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.704781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.705530 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.705531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.706290 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.706291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.706366 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.706380 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 02:42:45.706463 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:42:45.706510 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":4,"thread_start_us":75,"threads_started":1}
I20260928 02:42:45.706612 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.702924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.9.130.1:36779)
0928 02:42:45.703113 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:45.703119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:45.703129 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:42:45.703172 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:45.703175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:45.703191 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:45.703268 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:45.703501 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.704663 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.705624 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.706215 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.706293 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.706351 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:45.706489 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 02:42:45.706531 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":107,"threads_started":1}
I20260928 02:42:45.707034 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.702616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.9.130.2:33873)
0928 02:42:45.703168 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:45.703173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:45.703184 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:42:45.703230 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:45.703232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:45.703248 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:45.703324 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:45.703539 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.704365 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.705845 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.706590 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.706654 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.706717 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:45.706903 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 02:42:45.706945 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":86,"threads_started":1}
I20260928 02:42:45.707033 29325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.702508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:33873 (local address 127.0.0.1:56896)
0928 02:42:45.702743 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:45.702787 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:45.702850 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:45.703407 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:45.703408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:45.703410 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:45.703460 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.703462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.704482 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.704483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.705737 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.705740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.706752 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.706753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.706814 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.706817 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:45.706872 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:42:45.706918 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":31,"thread_start_us":43,"threads_started":1}
I20260928 02:42:45.707044 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.702591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:36085 (local address 127.0.0.1:40906)
0928 02:42:45.702789 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:45.702800 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:45.702853 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:45.703326 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:45.703330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:45.703332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:45.703397 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.703399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.704768 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.704769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.705885 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.705887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.706782 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.706783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.706841 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.706845 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:42:45.706874 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 02:42:45.706924 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":34,"thread_start_us":75,"threads_started":1}
I20260928 02:42:45.707464 29328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.702815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.9.130.3:36085)
0928 02:42:45.703158 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:45.703162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:45.703170 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:42:45.703200 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:45.703202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:45.703219 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:45.703478 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:45.703549 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.704650 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.705964 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.706682 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.707218 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.707288 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:45.707323 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 02:42:45.707363 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":78,"threads_started":1}
I20260928 02:42:45.707634 28979 tablet_service.cc:1467] Processing CreateTablet for tablet cce084ac21284c3bb23659c79518c448 (DEFAULT_TABLE table=test-table [id=2b72fdcad7884e30ab37824fb5166202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:45.707736 29113 tablet_service.cc:1467] Processing CreateTablet for tablet cce084ac21284c3bb23659c79518c448 (DEFAULT_TABLE table=test-table [id=2b72fdcad7884e30ab37824fb5166202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:45.708019 28979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce084ac21284c3bb23659c79518c448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:45.708027 29113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce084ac21284c3bb23659c79518c448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:45.708180 29247 tablet_service.cc:1467] Processing CreateTablet for tablet cce084ac21284c3bb23659c79518c448 (DEFAULT_TABLE table=test-table [id=2b72fdcad7884e30ab37824fb5166202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:45.708513 29247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce084ac21284c3bb23659c79518c448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:45.710973 29330 tablet_bootstrap.cc:492] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6: Bootstrap starting.
I20260928 02:42:45.711280 29331 tablet_bootstrap.cc:492] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b: Bootstrap starting.
I20260928 02:42:45.711833 29330 tablet_bootstrap.cc:654] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:45.711974 29331 tablet_bootstrap.cc:654] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:45.711971 29332 tablet_bootstrap.cc:492] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e: Bootstrap starting.
I20260928 02:42:45.712481 29330 log.cc:824] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:45.712535 29332 tablet_bootstrap.cc:654] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:45.712581 29331 log.cc:824] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:45.713109 29332 log.cc:824] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:45.713279 29330 tablet_bootstrap.cc:492] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6: No bootstrap required, opened a new log
I20260928 02:42:45.713390 29330 ts_tablet_manager.cc:1397] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:42:45.713795 29332 tablet_bootstrap.cc:492] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e: No bootstrap required, opened a new log
I20260928 02:42:45.713876 29332 ts_tablet_manager.cc:1397] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:42:45.713876 29331 tablet_bootstrap.cc:492] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b: No bootstrap required, opened a new log
I20260928 02:42:45.713935 29331 ts_tablet_manager.cc:1397] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:42:45.715359 29330 raft_consensus.cc:348] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee7544de0464e26a93d4bb3555ea90e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33873 } } peers { permanent_uuid: "f123b460a85d4757a2bc3ea72d29813b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 36779 } } peers { permanent_uuid: "ee2fecb0a8064ed493eeded534f39bd6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36085 } }
I20260928 02:42:45.715408 29332 raft_consensus.cc:348] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee7544de0464e26a93d4bb3555ea90e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33873 } } peers { permanent_uuid: "f123b460a85d4757a2bc3ea72d29813b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 36779 } } peers { permanent_uuid: "ee2fecb0a8064ed493eeded534f39bd6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36085 } }
I20260928 02:42:45.715546 29330 raft_consensus.cc:374] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:45.715569 29332 raft_consensus.cc:374] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:45.715586 29330 raft_consensus.cc:729] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2fecb0a8064ed493eeded534f39bd6, State: Initialized, Role: FOLLOWER
I20260928 02:42:45.715613 29332 raft_consensus.cc:729] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee7544de0464e26a93d4bb3555ea90e, State: Initialized, Role: FOLLOWER
I20260928 02:42:45.715708 29330 consensus_queue.cc:260] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee7544de0464e26a93d4bb3555ea90e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33873 } } peers { permanent_uuid: "f123b460a85d4757a2bc3ea72d29813b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 36779 } } peers { permanent_uuid: "ee2fecb0a8064ed493eeded534f39bd6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36085 } }
I20260928 02:42:45.715742 29332 consensus_queue.cc:260] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee7544de0464e26a93d4bb3555ea90e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33873 } } peers { permanent_uuid: "f123b460a85d4757a2bc3ea72d29813b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 36779 } } peers { permanent_uuid: "ee2fecb0a8064ed493eeded534f39bd6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36085 } }
I20260928 02:42:45.716010 29330 ts_tablet_manager.cc:1428] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 02:42:45.716042 29332 ts_tablet_manager.cc:1428] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:42:45.716063 29179 heartbeater.cc:503] Master 127.9.130.62:37159 was elected leader, sending a full tablet report...
I20260928 02:42:45.716133 29313 heartbeater.cc:503] Master 127.9.130.62:37159 was elected leader, sending a full tablet report...
I20260928 02:42:45.715940 29331 raft_consensus.cc:348] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee7544de0464e26a93d4bb3555ea90e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33873 } } peers { permanent_uuid: "f123b460a85d4757a2bc3ea72d29813b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 36779 } } peers { permanent_uuid: "ee2fecb0a8064ed493eeded534f39bd6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36085 } }
I20260928 02:42:45.716403 29331 raft_consensus.cc:374] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:45.716457 29331 raft_consensus.cc:729] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f123b460a85d4757a2bc3ea72d29813b, State: Initialized, Role: FOLLOWER
I20260928 02:42:45.716564 29331 consensus_queue.cc:260] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee7544de0464e26a93d4bb3555ea90e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33873 } } peers { permanent_uuid: "f123b460a85d4757a2bc3ea72d29813b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 36779 } } peers { permanent_uuid: "ee2fecb0a8064ed493eeded534f39bd6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36085 } }
I20260928 02:42:45.716830 29331 ts_tablet_manager.cc:1428] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:42:45.716977 29045 heartbeater.cc:503] Master 127.9.130.62:37159 was elected leader, sending a full tablet report...
I20260928 02:42:45.719939 29338 raft_consensus.cc:482] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:42:45.720007 29338 raft_consensus.cc:504] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee7544de0464e26a93d4bb3555ea90e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33873 } } peers { permanent_uuid: "f123b460a85d4757a2bc3ea72d29813b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 36779 } } peers { permanent_uuid: "ee2fecb0a8064ed493eeded534f39bd6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36085 } }
I20260928 02:42:45.720297 29338 leader_election.cc:288] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ee7544de0464e26a93d4bb3555ea90e (127.9.130.2:33873), ee2fecb0a8064ed493eeded534f39bd6 (127.9.130.3:36085)
I20260928 02:42:45.724110 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.720524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:55929 (local address 127.9.130.2:33873)
0928 02:42:45.720563 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:45.720565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:45.720662 (+ 97us) server_negotiation.cc:407] Connection header received
0928 02:42:45.720714 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:45.720715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:45.720726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:45.720790 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:45.720979 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.721690 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.723166 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.723741 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.723853 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.723861 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:45.724013 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:42:45.724049 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:42:45.724110 29328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.720543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:35277 (local address 127.9.130.3:36085)
0928 02:42:45.720586 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:45.720588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:45.720680 (+ 92us) server_negotiation.cc:407] Connection header received
0928 02:42:45.720737 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:45.720738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:45.720748 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:45.720798 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:45.720979 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.721704 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.723166 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.723751 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.723851 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.723861 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:45.724014 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 02:42:45.724050 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:42:45.724133 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.720470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:33873 (local address 127.9.130.1:55929)
0928 02:42:45.720591 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:45.720603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:45.720658 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:45.720803 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:45.720804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:45.720805 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:45.720903 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.720905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.721789 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.721790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.723075 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.723077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.723857 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.723857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.723926 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.723933 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:42:45.723997 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:42:45.724044 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":95}
I20260928 02:42:45.724138 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.720470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:36085 (local address 127.9.130.1:35277)
0928 02:42:45.720619 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:45.720627 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:45.720676 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:45.720803 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:45.720804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:45.720805 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:45.720903 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.720905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.721789 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.721790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.723075 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:45.723077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.723850 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.723851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:45.723926 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.723933 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:42:45.723994 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:42:45.724042 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":84,"thread_start_us":73,"threads_started":1}
I20260928 02:42:45.724560 29267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce084ac21284c3bb23659c79518c448" candidate_uuid: "f123b460a85d4757a2bc3ea72d29813b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2fecb0a8064ed493eeded534f39bd6" is_pre_election: true
I20260928 02:42:45.724692 29267 raft_consensus.cc:2454] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f123b460a85d4757a2bc3ea72d29813b in term 0.
I20260928 02:42:45.724858 29133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce084ac21284c3bb23659c79518c448" candidate_uuid: "f123b460a85d4757a2bc3ea72d29813b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee7544de0464e26a93d4bb3555ea90e" is_pre_election: true
I20260928 02:42:45.724895 28933 leader_election.cc:302] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee2fecb0a8064ed493eeded534f39bd6, f123b460a85d4757a2bc3ea72d29813b; no voters:
I20260928 02:42:45.724951 29133 raft_consensus.cc:2454] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f123b460a85d4757a2bc3ea72d29813b in term 0.
I20260928 02:42:45.725046 29338 raft_consensus.cc:2781] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:42:45.725108 29338 raft_consensus.cc:482] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:42:45.725134 29338 raft_consensus.cc:3037] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:45.725761 29338 raft_consensus.cc:504] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee7544de0464e26a93d4bb3555ea90e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33873 } } peers { permanent_uuid: "f123b460a85d4757a2bc3ea72d29813b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 36779 } } peers { permanent_uuid: "ee2fecb0a8064ed493eeded534f39bd6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36085 } }
I20260928 02:42:45.725894 29338 leader_election.cc:288] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [CANDIDATE]: Term 1 election: Requested vote from peers 6ee7544de0464e26a93d4bb3555ea90e (127.9.130.2:33873), ee2fecb0a8064ed493eeded534f39bd6 (127.9.130.3:36085)
I20260928 02:42:45.726016 29133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce084ac21284c3bb23659c79518c448" candidate_uuid: "f123b460a85d4757a2bc3ea72d29813b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee7544de0464e26a93d4bb3555ea90e"
I20260928 02:42:45.726034 29267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce084ac21284c3bb23659c79518c448" candidate_uuid: "f123b460a85d4757a2bc3ea72d29813b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2fecb0a8064ed493eeded534f39bd6"
I20260928 02:42:45.726092 29267 raft_consensus.cc:3037] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:45.726092 29133 raft_consensus.cc:3037] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:45.726907 29267 raft_consensus.cc:2454] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f123b460a85d4757a2bc3ea72d29813b in term 1.
I20260928 02:42:45.726907 29133 raft_consensus.cc:2454] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f123b460a85d4757a2bc3ea72d29813b in term 1.
I20260928 02:42:45.727063 28935 leader_election.cc:302] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ee7544de0464e26a93d4bb3555ea90e, f123b460a85d4757a2bc3ea72d29813b; no voters:
I20260928 02:42:45.727161 29338 raft_consensus.cc:2781] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:45.727329 29338 raft_consensus.cc:686] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [term 1 LEADER]: Becoming Leader. State: Replica: f123b460a85d4757a2bc3ea72d29813b, State: Running, Role: LEADER
I20260928 02:42:45.727427 29338 consensus_queue.cc:237] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee7544de0464e26a93d4bb3555ea90e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33873 } } peers { permanent_uuid: "f123b460a85d4757a2bc3ea72d29813b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 36779 } } peers { permanent_uuid: "ee2fecb0a8064ed493eeded534f39bd6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36085 } }
I20260928 02:42:45.728168 28853 catalog_manager.cc:5495] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b reported cstate change: term changed from 0 to 1, leader changed from <none> to f123b460a85d4757a2bc3ea72d29813b (127.9.130.1). New cstate: current_term: 1 leader_uuid: "f123b460a85d4757a2bc3ea72d29813b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee7544de0464e26a93d4bb3555ea90e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f123b460a85d4757a2bc3ea72d29813b" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 36779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee2fecb0a8064ed493eeded534f39bd6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36085 } health_report { overall_health: UNKNOWN } } }
I20260928 02:42:45.737263 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.735080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.9.130.1:36779)
0928 02:42:45.735130 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:45.735132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:45.735140 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:42:45.735170 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:45.735171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:45.735195 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:45.735254 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:42:45.735359 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.735801 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.736573 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.736816 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.736874 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.736998 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:42:45.737107 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:42:45.737185 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:45.737199 (+ 14us) server_negotiation.cc:299] Negotiation successful
0928 02:42:45.737215 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:42:45.738845 29267 raft_consensus.cc:1261] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6 [term 1 FOLLOWER]: Refusing update from remote peer f123b460a85d4757a2bc3ea72d29813b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:45.738864 29133 raft_consensus.cc:1261] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e [term 1 FOLLOWER]: Refusing update from remote peer f123b460a85d4757a2bc3ea72d29813b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:45.739128 29338 consensus_queue.cc:1035] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee2fecb0a8064ed493eeded534f39bd6" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 36085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:45.739248 29340 consensus_queue.cc:1035] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ee7544de0464e26a93d4bb3555ea90e" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 33873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:45.741472 29343 mvcc.cc:205] Tried to move back new op lower bound from 7334147546064424960 to 7334147546020470784. Current Snapshot: MvccSnapshot[applied={T|T < 7334147546064424960}]
I20260928 02:42:45.741688 29344 mvcc.cc:205] Tried to move back new op lower bound from 7334147546064424960 to 7334147546020470784. Current Snapshot: MvccSnapshot[applied={T|T < 7334147546064424960}]
I20260928 02:42:45.743614 28854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:45.743696 28854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:45.744400 28854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2ebb303ab254d20b88474026b6cfd35: Sending DeleteTablet for 3 replicas of tablet cce084ac21284c3bb23659c79518c448
I20260928 02:42:45.744688 29113 tablet_service.cc:1514] Processing DeleteTablet for tablet cce084ac21284c3bb23659c79518c448 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:45 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:56896
I20260928 02:42:45.744714 28979 tablet_service.cc:1514] Processing DeleteTablet for tablet cce084ac21284c3bb23659c79518c448 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:45 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:42034
I20260928 02:42:45.744721 29247 tablet_service.cc:1514] Processing DeleteTablet for tablet cce084ac21284c3bb23659c79518c448 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:45 UTC) from {username='kudu', principal='kudu/127.9.130.62@KRBTEST.COM'} at 127.0.0.1:40906
I20260928 02:42:45.744916 29353 tablet_replica.cc:330] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b: stopping tablet replica
I20260928 02:42:45.745018 29353 raft_consensus.cc:2229] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:42:45.745112 29353 raft_consensus.cc:2258] T cce084ac21284c3bb23659c79518c448 P f123b460a85d4757a2bc3ea72d29813b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:45.745257 29352 tablet_replica.cc:330] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e: stopping tablet replica
I20260928 02:42:45.745282 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 28916
I20260928 02:42:45.745349 29352 raft_consensus.cc:2229] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:42:45.746093 29352 raft_consensus.cc:2258] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:45.746517 29352 ts_tablet_manager.cc:1905] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:45.746711 29354 tablet_replica.cc:330] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6: stopping tablet replica
I20260928 02:42:45.746860 29354 raft_consensus.cc:2229] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:42:45.746974 29354 raft_consensus.cc:2258] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:45.747406 29354 ts_tablet_manager.cc:1905] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:45.748260 29352 ts_tablet_manager.cc:1918] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:45.748338 29352 log.cc:1196] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-1/wal/wals/cce084ac21284c3bb23659c79518c448
I20260928 02:42:45.748595 29352 ts_tablet_manager.cc:1939] T cce084ac21284c3bb23659c79518c448 P 6ee7544de0464e26a93d4bb3555ea90e: Deleting consensus metadata
I20260928 02:42:45.748986 28841 catalog_manager.cc:4841] TS 6ee7544de0464e26a93d4bb3555ea90e (127.9.130.2:33873): tablet cce084ac21284c3bb23659c79518c448 (table test-table [id=2b72fdcad7884e30ab37824fb5166202]) successfully deleted
I20260928 02:42:45.749305 29354 ts_tablet_manager.cc:1918] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:45.749378 29354 log.cc:1196] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/cce084ac21284c3bb23659c79518c448
I20260928 02:42:45.749626 29354 ts_tablet_manager.cc:1939] T cce084ac21284c3bb23659c79518c448 P ee2fecb0a8064ed493eeded534f39bd6: Deleting consensus metadata
I20260928 02:42:45.749989 28839 catalog_manager.cc:4841] TS ee2fecb0a8064ed493eeded534f39bd6 (127.9.130.3:36085): tablet cce084ac21284c3bb23659c79518c448 (table test-table [id=2b72fdcad7884e30ab37824fb5166202]) successfully deleted
W20260928 02:42:45.751799 28841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:36779 (error 108)
I20260928 02:42:45.752041 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 29050
I20260928 02:42:45.752647 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.752169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:36779 (local address 127.0.0.1:42050)
0928 02:42:45.752570 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:45.752607 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:36779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381}
W20260928 02:42:45.752833 28841 catalog_manager.cc:4584] TS f123b460a85d4757a2bc3ea72d29813b (127.9.130.1:36779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cce084ac21284c3bb23659c79518c448: Network error: Client connection negotiation failed: client connection to 127.9.130.1:36779: connect: Connection refused (error 111)
I20260928 02:42:45.757141 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 29184
I20260928 02:42:45.762166 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 28822
2026-09-28T02:42:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1879 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T02:42:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:45Z Disabled control of system clock
I20260928 02:42:45.789713 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:36583
--webserver_interface=127.9.130.62
--webserver_port=0
--builtin_ntp_servers=127.9.130.20:38573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:36583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:42:45.871805 29359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:45.872087 29359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:45.872107 29359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:45.873366 29359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:45.873406 29359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:45.873420 29359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:45.873432 29359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:45.873462 29359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:45.874814 29359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:36583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:36583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:45.875056 29359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:45.875367 29359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:45.880251 29364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:45.880272 29365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:45.880251 29367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:45.880625 29359 server_base.cc:1034] running on GCE node
I20260928 02:42:45.880884 29359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:45.881168 29359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:45.882328 29359 hybrid_clock.cc:648] HybridClock initialized: now 1790563365882320 us; error 40 us; skew 500 ppm
I20260928 02:42:45.883599 29359 webserver.cc:460] Webserver started at http://127.9.130.62:41703/ using document root <none> and password file <none>
I20260928 02:42:45.883885 29359 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:45.883965 29359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:45.884099 29359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:45.885125 29359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "f8c6c979a03e41f380e09807a34ad853"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "63550cb8884ae72bb2a6b8c442aa2d19"
server_key_iv: "f6291d670647024ad98bbc0bb57bae25"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.885497 29359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "f8c6c979a03e41f380e09807a34ad853"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "63550cb8884ae72bb2a6b8c442aa2d19"
server_key_iv: "f6291d670647024ad98bbc0bb57bae25"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.886754 29359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:42:45.887470 29373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:45.887642 29359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:42:45.887739 29359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "f8c6c979a03e41f380e09807a34ad853"
format_stamp: "Formatted at 2026-09-28 02:42:45 on dist-test-slave-kkrh"
server_key: "63550cb8884ae72bb2a6b8c442aa2d19"
server_key_iv: "f6291d670647024ad98bbc0bb57bae25"
server_key_version: "encryptionkey@0"
I20260928 02:42:45.887822 29359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:45.904069 29359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:45.904337 29359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:45.904493 29359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:45.908142 29359 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:36583
I20260928 02:42:45.908174 29425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:36583 every 8 connection(s)
I20260928 02:42:45.908500 29359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:45.909420 29426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:45.911514 29426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853: Bootstrap starting.
I20260928 02:42:45.912094 29426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:45.912380 29426 log.cc:824] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:45.913007 29426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853: No bootstrap required, opened a new log
I20260928 02:42:45.913730 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 29359
I20260928 02:42:45.913846 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:45.914034 9736 external_mini_cluster.cc:1428] Setting key 497f2692a260cd01988c92ee68800733
I20260928 02:42:45.915077 29426 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c6c979a03e41f380e09807a34ad853" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36583 } }
I20260928 02:42:45.915239 29426 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:45.915274 29426 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c6c979a03e41f380e09807a34ad853, State: Initialized, Role: FOLLOWER
I20260928 02:42:45.915397 29426 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c6c979a03e41f380e09807a34ad853" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36583 } }
I20260928 02:42:45.915494 29426 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:45.915532 29426 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:45.915570 29426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:45.916285 29426 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c6c979a03e41f380e09807a34ad853" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36583 } }
I20260928 02:42:45.916388 29426 leader_election.cc:302] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8c6c979a03e41f380e09807a34ad853; no voters:
I20260928 02:42:45.916527 29426 leader_election.cc:288] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:45.916597 29431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:45.916725 29431 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [term 1 LEADER]: Becoming Leader. State: Replica: f8c6c979a03e41f380e09807a34ad853, State: Running, Role: LEADER
I20260928 02:42:45.916745 29426 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:45.916813 29431 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c6c979a03e41f380e09807a34ad853" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36583 } }
I20260928 02:42:45.917172 29431 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8c6c979a03e41f380e09807a34ad853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c6c979a03e41f380e09807a34ad853" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36583 } } }
I20260928 02:42:45.917248 29431 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:45.917383 29431 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8c6c979a03e41f380e09807a34ad853. Latest consensus state: current_term: 1 leader_uuid: "f8c6c979a03e41f380e09807a34ad853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c6c979a03e41f380e09807a34ad853" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36583 } } }
I20260928 02:42:45.917446 29431 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:45.917757 29439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:45.918234 29439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:45.920054 29439 catalog_manager.cc:1347] Generated new cluster ID: 18d8e55d2c034f85b836227eca43d99d
I20260928 02:42:45.920111 29439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:45.920677 29430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:45.914521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.9.130.62:36583)
0928 02:42:45.914749 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:45.914760 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:45.914796 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:42:45.915041 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:45.915049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:45.915124 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:45.915274 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:45.915528 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.917389 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.918544 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:45.919075 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:45.919195 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:45.919932 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:45.919936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:45.919937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:45.920165 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:45.920208 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:45.920223 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:45.920294 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:45.920418 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 02:42:45.920489 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260928 02:42:45.927973 29439 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:45.928455 29439 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:45.935113 29439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853: Generated new TSK 0
I20260928 02:42:45.935268 29439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:45.940270 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.1:0
--local_ip_for_outbound_sockets=127.9.130.1
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:36583
--builtin_ntp_servers=127.9.130.20:38573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:42:46.020730 29450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:46.020924 29450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:46.020946 29450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:46.022087 29450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:46.022125 29450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:46.022150 29450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.1
I20260928 02:42:46.023465 29450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.130.1
--webserver_port=0
--tserver_master_addrs=127.9.130.62:36583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.1
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:46.023686 29450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:46.024017 29450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:46.028877 29455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:46.028887 29458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:46.028950 29456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:46.029233 29450 server_base.cc:1034] running on GCE node
I20260928 02:42:46.029454 29450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:46.029690 29450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:46.030860 29450 hybrid_clock.cc:648] HybridClock initialized: now 1790563366030844 us; error 28 us; skew 500 ppm
I20260928 02:42:46.032274 29450 webserver.cc:460] Webserver started at http://127.9.130.1:34123/ using document root <none> and password file <none>
I20260928 02:42:46.032521 29450 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:46.032591 29450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:46.032720 29450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:46.034144 29450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/data/instance:
uuid: "56efd46f3eb44f2086f7480855959bce"
format_stamp: "Formatted at 2026-09-28 02:42:46 on dist-test-slave-kkrh"
server_key: "19073fb764066cf21223f5b8e525d949"
server_key_iv: "4b24acb93924c156d616ea8d20855644"
server_key_version: "encryptionkey@0"
I20260928 02:42:46.034621 29450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance:
uuid: "56efd46f3eb44f2086f7480855959bce"
format_stamp: "Formatted at 2026-09-28 02:42:46 on dist-test-slave-kkrh"
server_key: "19073fb764066cf21223f5b8e525d949"
server_key_iv: "4b24acb93924c156d616ea8d20855644"
server_key_version: "encryptionkey@0"
I20260928 02:42:46.036245 29450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260928 02:42:46.037042 29464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:46.037235 29450 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:46.037325 29450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/wal
uuid: "56efd46f3eb44f2086f7480855959bce"
format_stamp: "Formatted at 2026-09-28 02:42:46 on dist-test-slave-kkrh"
server_key: "19073fb764066cf21223f5b8e525d949"
server_key_iv: "4b24acb93924c156d616ea8d20855644"
server_key_version: "encryptionkey@0"
I20260928 02:42:46.037390 29450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:46.073271 29450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:46.073604 29450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:46.073717 29450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:46.073930 29450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:46.074190 29450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:46.074219 29450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:46.074240 29450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:46.074254 29450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:46.080026 29450 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.1:39617
I20260928 02:42:46.080108 29577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.1:39617 every 8 connection(s)
I20260928 02:42:46.080417 29450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/data/info.pb
I20260928 02:42:46.085318 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 29450
I20260928 02:42:46.085400 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-0/wal/instance
I20260928 02:42:46.085601 9736 external_mini_cluster.cc:1428] Setting key 332d159d4e2c46d83809df92cf0ff363
I20260928 02:42:46.085765 29430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.081156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.1:34857 (local address 127.9.130.62:36583)
0928 02:42:46.081196 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:46.081198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:46.081478 (+ 280us) server_negotiation.cc:407] Connection header received
0928 02:42:46.082019 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:46.082020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:46.082035 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:46.082086 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:46.082800 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.083296 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.084345 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.084545 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.084618 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.085491 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:46.085492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:46.085492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:46.085497 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:46.085512 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:46.085518 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:46.085547 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:46.085658 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 02:42:46.085688 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:42:46.085983 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.081169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:36583 (local address 127.9.130.1:34857)
0928 02:42:46.081361 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:46.081394 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:46.081909 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:46.082151 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:46.082154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:46.082164 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:46.082707 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.082712 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.083402 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.083403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.084251 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.084254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.084638 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.084639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.084765 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.085311 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:46.085311 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:46.085340 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:46.085382 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:46.085580 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:46.085585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:46.085588 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:46.085659 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:42:46.085754 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260928 02:42:46.086460 29578 heartbeater.cc:347] Connected to a master server at 127.9.130.62:36583
I20260928 02:42:46.086535 29578 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:46.086666 29578 heartbeater.cc:511] Master 127.9.130.62:36583 requested a full tablet report, sending...
I20260928 02:42:46.086953 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.2:0
--local_ip_for_outbound_sockets=127.9.130.2
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:36583
--builtin_ntp_servers=127.9.130.20:38573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:42:46.087172 29390 ts_manager.cc:194] Registered new tserver with Master: 56efd46f3eb44f2086f7480855959bce (127.9.130.1:39617)
I20260928 02:42:46.088100 29390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.130.1:34857
W20260928 02:42:46.166113 29581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:46.166311 29581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:46.166332 29581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:46.167455 29581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:46.167492 29581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:46.167516 29581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.2
I20260928 02:42:46.168821 29581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.130.2
--webserver_port=0
--tserver_master_addrs=127.9.130.62:36583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.2
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:46.169059 29581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:46.169346 29581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:46.174324 29587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:46.174310 29586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:46.174343 29589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:46.174506 29581 server_base.cc:1034] running on GCE node
I20260928 02:42:46.174738 29581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:46.174969 29581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:46.176142 29581 hybrid_clock.cc:648] HybridClock initialized: now 1790563366176127 us; error 43 us; skew 500 ppm
I20260928 02:42:46.177474 29581 webserver.cc:460] Webserver started at http://127.9.130.2:40903/ using document root <none> and password file <none>
I20260928 02:42:46.177716 29581 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:46.177788 29581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:46.177940 29581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:46.178949 29581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/data/instance:
uuid: "cb631c8eaaf04e749c05dbe933f56514"
format_stamp: "Formatted at 2026-09-28 02:42:46 on dist-test-slave-kkrh"
server_key: "91e2aa1be36fa5d4dbdd81c4fcc285be"
server_key_iv: "5ff0e06b0061fcac142b82f1ddd2e801"
server_key_version: "encryptionkey@0"
I20260928 02:42:46.179311 29581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance:
uuid: "cb631c8eaaf04e749c05dbe933f56514"
format_stamp: "Formatted at 2026-09-28 02:42:46 on dist-test-slave-kkrh"
server_key: "91e2aa1be36fa5d4dbdd81c4fcc285be"
server_key_iv: "5ff0e06b0061fcac142b82f1ddd2e801"
server_key_version: "encryptionkey@0"
I20260928 02:42:46.180648 29581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:42:46.181475 29595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:46.181643 29581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:42:46.181740 29581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/wal
uuid: "cb631c8eaaf04e749c05dbe933f56514"
format_stamp: "Formatted at 2026-09-28 02:42:46 on dist-test-slave-kkrh"
server_key: "91e2aa1be36fa5d4dbdd81c4fcc285be"
server_key_iv: "5ff0e06b0061fcac142b82f1ddd2e801"
server_key_version: "encryptionkey@0"
I20260928 02:42:46.181823 29581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:46.193970 29581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:46.194231 29581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:46.194386 29581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:46.194585 29581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:46.194882 29581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:46.194936 29581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:46.194974 29581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:46.195000 29581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:46.200744 29581 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.2:45763
I20260928 02:42:46.200802 29708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.2:45763 every 8 connection(s)
I20260928 02:42:46.201092 29581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/data/info.pb
I20260928 02:42:46.201772 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 29581
I20260928 02:42:46.201913 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-1/wal/instance
I20260928 02:42:46.202131 9736 external_mini_cluster.cc:1428] Setting key bbc88031c9458ffef1f7abeed6e8af94
I20260928 02:42:46.203153 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.130.3:0
--local_ip_for_outbound_sockets=127.9.130.3
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:36583
--builtin_ntp_servers=127.9.130.20:38573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:42:46.206739 29430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.201765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.2:40955 (local address 127.9.130.62:36583)
0928 02:42:46.201851 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:46.201854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:46.202087 (+ 233us) server_negotiation.cc:407] Connection header received
0928 02:42:46.202636 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:46.202637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:46.202649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:46.202712 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:46.203575 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.204110 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.205196 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.205524 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.205596 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.206421 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:46.206423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:46.206423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:46.206428 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:46.206445 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:46.206451 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:46.206511 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:46.206598 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:42:46.206627 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:42:46.206851 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.201721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:36583 (local address 127.9.130.2:40955)
0928 02:42:46.201959 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:46.202001 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:46.202542 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:46.202776 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:46.202780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:46.202790 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:46.203482 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.203490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.204196 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.204197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.205106 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.205109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.205610 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.205611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.205713 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.206279 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:46.206280 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:46.206309 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:46.206330 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:46.206526 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:46.206531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:46.206534 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:46.206583 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:42:46.206656 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260928 02:42:46.207311 29709 heartbeater.cc:347] Connected to a master server at 127.9.130.62:36583
I20260928 02:42:46.207407 29709 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:46.207609 29709 heartbeater.cc:511] Master 127.9.130.62:36583 requested a full tablet report, sending...
I20260928 02:42:46.208031 29390 ts_manager.cc:194] Registered new tserver with Master: cb631c8eaaf04e749c05dbe933f56514 (127.9.130.2:45763)
I20260928 02:42:46.208559 29390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.130.2:40955
W20260928 02:42:46.282684 29712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:46.282873 29712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:46.282894 29712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:46.283959 29712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:46.283990 29712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:46.284014 29712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.130.3
I20260928 02:42:46.285403 29712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:38573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.130.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.130.3
--webserver_port=0
--tserver_master_addrs=127.9.130.62:36583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.130.3
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:46.285604 29712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:46.285838 29712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:46.290527 29717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:46.290552 29718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:46.290565 29720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:46.290791 29712 server_base.cc:1034] running on GCE node
I20260928 02:42:46.291039 29712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:46.291273 29712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:46.292404 29712 hybrid_clock.cc:648] HybridClock initialized: now 1790563366292385 us; error 35 us; skew 500 ppm
I20260928 02:42:46.293632 29712 webserver.cc:460] Webserver started at http://127.9.130.3:34115/ using document root <none> and password file <none>
I20260928 02:42:46.293877 29712 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:46.293924 29712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:46.294085 29712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:46.295070 29712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/data/instance:
uuid: "b79cd424e129470888e06bdb4daa0af2"
format_stamp: "Formatted at 2026-09-28 02:42:46 on dist-test-slave-kkrh"
server_key: "5e65d4d78b0e0c95952b568af1edf0e6"
server_key_iv: "a4e41574d201c02830cb97b2c9915d1c"
server_key_version: "encryptionkey@0"
I20260928 02:42:46.295406 29712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance:
uuid: "b79cd424e129470888e06bdb4daa0af2"
format_stamp: "Formatted at 2026-09-28 02:42:46 on dist-test-slave-kkrh"
server_key: "5e65d4d78b0e0c95952b568af1edf0e6"
server_key_iv: "a4e41574d201c02830cb97b2c9915d1c"
server_key_version: "encryptionkey@0"
I20260928 02:42:46.296689 29712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:42:46.297340 29726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:46.297492 29712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:42:46.297586 29712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/wal
uuid: "b79cd424e129470888e06bdb4daa0af2"
format_stamp: "Formatted at 2026-09-28 02:42:46 on dist-test-slave-kkrh"
server_key: "5e65d4d78b0e0c95952b568af1edf0e6"
server_key_iv: "a4e41574d201c02830cb97b2c9915d1c"
server_key_version: "encryptionkey@0"
I20260928 02:42:46.297662 29712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:46.326323 29712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:46.326622 29712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:46.326777 29712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:46.326999 29712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:46.327311 29712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:46.327364 29712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:46.327406 29712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:46.327433 29712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:46.333077 29712 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.3:43037
I20260928 02:42:46.333127 29839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.3:43037 every 8 connection(s)
I20260928 02:42:46.333424 29712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/data/info.pb
I20260928 02:42:46.338397 29430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.334138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:35619 (local address 127.9.130.62:36583)
0928 02:42:46.334197 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:46.334199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:46.334413 (+ 214us) server_negotiation.cc:407] Connection header received
0928 02:42:46.334867 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:46.334868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:46.334878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:46.334937 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:46.335551 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.336113 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.337057 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.337237 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.337297 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.338057 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:46.338058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:46.338059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:46.338062 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:46.338075 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:46.338079 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:46.338123 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:46.338270 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:42:46.338304 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:42:46.338398 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 29712
I20260928 02:42:46.338505 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/wal/instance
I20260928 02:42:46.338606 29842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.334091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:36583 (local address 127.9.130.3:35619)
0928 02:42:46.334283 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:46.334324 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:46.334788 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:46.334987 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:46.334990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:46.334998 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:46.335477 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.335480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.336186 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.336187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.336972 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.336974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.337325 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.337326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.337434 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.337916 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:46.337916 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:46.337948 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:46.337969 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:46.338155 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:46.338160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:46.338163 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:46.338243 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:42:46.338323 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260928 02:42:46.338699 9736 external_mini_cluster.cc:1428] Setting key 744ffefda12426bfbf017ca0dbc7dacc
I20260928 02:42:46.339241 29840 heartbeater.cc:347] Connected to a master server at 127.9.130.62:36583
I20260928 02:42:46.339380 29840 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:46.339596 29840 heartbeater.cc:511] Master 127.9.130.62:36583 requested a full tablet report, sending...
I20260928 02:42:46.340003 29389 ts_manager.cc:194] Registered new tserver with Master: b79cd424e129470888e06bdb4daa0af2 (127.9.130.3:43037)
I20260928 02:42:46.340358 29389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.130.3:35619
I20260928 02:42:46.350608 9736 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:42:46.354995 29430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.352049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.9.130.62:36583)
0928 02:42:46.352176 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:46.352178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:46.352218 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:42:46.352289 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:46.352290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:46.352303 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:46.352375 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:46.352543 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.353026 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.353766 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.353960 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.354035 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.354738 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:46.354739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:46.354740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:46.354744 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:46.354765 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:46.354769 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:46.354830 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:46.354883 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 02:42:46.354908 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:42:46.356815 29389 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:42:46.358376 29389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:42:46.364615 29849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.360568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:45763 (local address 127.0.0.1:38814)
0928 02:42:46.360954 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:46.360964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:46.361019 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:46.361330 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:46.361331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:46.361333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:46.361385 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.361387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.362304 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.362306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.363481 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.363483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.364336 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.364337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.364406 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.364418 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:42:46.364494 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:42:46.364542 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":4,"thread_start_us":70,"threads_started":1}
I20260928 02:42:46.364626 29851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.360745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.9.130.2:45763)
0928 02:42:46.361144 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:46.361165 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:46.361174 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:42:46.361214 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:46.361216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:46.361232 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:46.361464 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:46.361525 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.362214 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.363577 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.364248 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.364338 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.364432 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:46.364507 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:42:46.364546 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":75,"threads_started":1}
I20260928 02:42:46.364948 29852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.360772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.3:43037 (local address 127.0.0.1:49784)
0928 02:42:46.360979 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:46.360988 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:46.361034 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:46.361355 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:46.361355 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:46.361357 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:46.361392 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.361393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.362725 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.362726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.363653 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.363656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.364678 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.364679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.364754 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.364760 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:42:46.364824 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:42:46.364862 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":2,"thread_start_us":73,"threads_started":1}
I20260928 02:42:46.365068 29850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.360650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.9.130.1:39617)
0928 02:42:46.360874 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:46.360879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:46.360890 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:42:46.361158 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:46.361162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:46.361182 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:46.361456 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:46.361548 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.362606 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.363796 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.364505 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.364561 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.364641 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:46.364944 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:42:46.364989 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
I20260928 02:42:46.365216 29853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.360980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.9.130.3:43037)
0928 02:42:46.361152 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:46.361158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:46.361167 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:42:46.361201 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:46.361203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:46.361219 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:46.361291 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:46.361468 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.362607 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.363738 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.364481 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.364559 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.364641 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:46.365070 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 02:42:46.365118 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260928 02:42:46.365257 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.360568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:39617 (local address 127.0.0.1:36452)
0928 02:42:46.360744 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:46.360805 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:46.360870 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:46.361306 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:46.361309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:46.361311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:46.361373 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.361375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.362696 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.362697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.363724 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.363726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.364798 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.364799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.364853 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.364855 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:42:46.364909 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:42:46.364940 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"spinlock_wait_cycles":9472,"thread_start_us":94,"threads_started":1}
I20260928 02:42:46.365517 29643 tablet_service.cc:1467] Processing CreateTablet for tablet 6402eca6ee034b1692306999ce5785fe (DEFAULT_TABLE table=test-table [id=6b2dbadf094c47a8aa101c05bb3253ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:46.365873 29643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6402eca6ee034b1692306999ce5785fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:46.365928 29512 tablet_service.cc:1467] Processing CreateTablet for tablet 6402eca6ee034b1692306999ce5785fe (DEFAULT_TABLE table=test-table [id=6b2dbadf094c47a8aa101c05bb3253ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:46.366235 29512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6402eca6ee034b1692306999ce5785fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:46.368420 29774 tablet_service.cc:1467] Processing CreateTablet for tablet 6402eca6ee034b1692306999ce5785fe (DEFAULT_TABLE table=test-table [id=6b2dbadf094c47a8aa101c05bb3253ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:46.368754 29774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6402eca6ee034b1692306999ce5785fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:46.369324 29854 tablet_bootstrap.cc:492] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514: Bootstrap starting.
I20260928 02:42:46.369481 29855 tablet_bootstrap.cc:492] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce: Bootstrap starting.
I20260928 02:42:46.370028 29855 tablet_bootstrap.cc:654] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:46.370023 29854 tablet_bootstrap.cc:654] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:46.370322 29855 log.cc:824] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:46.370368 29854 log.cc:824] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:46.370977 29855 tablet_bootstrap.cc:492] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce: No bootstrap required, opened a new log
I20260928 02:42:46.371055 29855 ts_tablet_manager.cc:1397] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:42:46.371650 29858 tablet_bootstrap.cc:492] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2: Bootstrap starting.
I20260928 02:42:46.372340 29858 tablet_bootstrap.cc:654] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:46.372632 29854 tablet_bootstrap.cc:492] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514: No bootstrap required, opened a new log
I20260928 02:42:46.372638 29858 log.cc:824] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:46.372701 29854 ts_tablet_manager.cc:1397] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:42:46.373221 29858 tablet_bootstrap.cc:492] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2: No bootstrap required, opened a new log
I20260928 02:42:46.373306 29858 ts_tablet_manager.cc:1397] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:42:46.373697 29855 raft_consensus.cc:348] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56efd46f3eb44f2086f7480855959bce" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 39617 } } peers { permanent_uuid: "b79cd424e129470888e06bdb4daa0af2" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43037 } } peers { permanent_uuid: "cb631c8eaaf04e749c05dbe933f56514" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45763 } }
I20260928 02:42:46.373875 29855 raft_consensus.cc:374] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:46.373914 29855 raft_consensus.cc:729] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56efd46f3eb44f2086f7480855959bce, State: Initialized, Role: FOLLOWER
I20260928 02:42:46.374047 29855 consensus_queue.cc:260] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56efd46f3eb44f2086f7480855959bce" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 39617 } } peers { permanent_uuid: "b79cd424e129470888e06bdb4daa0af2" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43037 } } peers { permanent_uuid: "cb631c8eaaf04e749c05dbe933f56514" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45763 } }
I20260928 02:42:46.374312 29855 ts_tablet_manager.cc:1428] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:42:46.374363 29578 heartbeater.cc:503] Master 127.9.130.62:36583 was elected leader, sending a full tablet report...
I20260928 02:42:46.375226 29854 raft_consensus.cc:348] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56efd46f3eb44f2086f7480855959bce" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 39617 } } peers { permanent_uuid: "b79cd424e129470888e06bdb4daa0af2" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43037 } } peers { permanent_uuid: "cb631c8eaaf04e749c05dbe933f56514" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45763 } }
I20260928 02:42:46.375381 29854 raft_consensus.cc:374] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:46.375433 29854 raft_consensus.cc:729] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb631c8eaaf04e749c05dbe933f56514, State: Initialized, Role: FOLLOWER
I20260928 02:42:46.375551 29854 consensus_queue.cc:260] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56efd46f3eb44f2086f7480855959bce" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 39617 } } peers { permanent_uuid: "b79cd424e129470888e06bdb4daa0af2" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43037 } } peers { permanent_uuid: "cb631c8eaaf04e749c05dbe933f56514" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45763 } }
I20260928 02:42:46.375671 29858 raft_consensus.cc:348] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56efd46f3eb44f2086f7480855959bce" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 39617 } } peers { permanent_uuid: "b79cd424e129470888e06bdb4daa0af2" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43037 } } peers { permanent_uuid: "cb631c8eaaf04e749c05dbe933f56514" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45763 } }
I20260928 02:42:46.375854 29854 ts_tablet_manager.cc:1428] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 02:42:46.375855 29858 raft_consensus.cc:374] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:46.375893 29709 heartbeater.cc:503] Master 127.9.130.62:36583 was elected leader, sending a full tablet report...
I20260928 02:42:46.375896 29858 raft_consensus.cc:729] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79cd424e129470888e06bdb4daa0af2, State: Initialized, Role: FOLLOWER
I20260928 02:42:46.376005 29858 consensus_queue.cc:260] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56efd46f3eb44f2086f7480855959bce" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 39617 } } peers { permanent_uuid: "b79cd424e129470888e06bdb4daa0af2" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43037 } } peers { permanent_uuid: "cb631c8eaaf04e749c05dbe933f56514" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45763 } }
I20260928 02:42:46.376248 29858 ts_tablet_manager.cc:1428] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 02:42:46.376369 29840 heartbeater.cc:503] Master 127.9.130.62:36583 was elected leader, sending a full tablet report...
I20260928 02:42:46.421653 29862 raft_consensus.cc:482] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:42:46.421770 29862 raft_consensus.cc:504] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56efd46f3eb44f2086f7480855959bce" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 39617 } } peers { permanent_uuid: "b79cd424e129470888e06bdb4daa0af2" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43037 } } peers { permanent_uuid: "cb631c8eaaf04e749c05dbe933f56514" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45763 } }
I20260928 02:42:46.422260 29862 leader_election.cc:288] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56efd46f3eb44f2086f7480855959bce (127.9.130.1:39617), cb631c8eaaf04e749c05dbe933f56514 (127.9.130.2:45763)
I20260928 02:42:46.425200 29842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.422394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:39617 (local address 127.9.130.3:57639)
0928 02:42:46.422457 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:46.422469 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:46.422508 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:46.422691 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:46.422693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:46.422695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:46.422767 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.422769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.423505 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.423506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.424388 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.424390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.424942 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.424943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.425011 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.425014 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:46.425066 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:42:46.425121 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 02:42:46.425200 29850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.422471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:57639 (local address 127.9.130.1:39617)
0928 02:42:46.422512 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:46.422514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:46.422522 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:42:46.422598 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:46.422599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:46.422613 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:46.422669 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:46.422868 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.423416 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.424474 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.424860 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.424914 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.424922 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:46.425087 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:42:46.425129 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:42:46.425237 29851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.422525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.130.3:52977 (local address 127.9.130.2:45763)
0928 02:42:46.422570 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:46.422572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:46.422672 (+ 100us) server_negotiation.cc:407] Connection header received
0928 02:42:46.422705 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:46.422706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:46.422714 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:46.422779 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:46.423035 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.423572 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.424532 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.424868 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.424959 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.424968 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:46.425126 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:42:46.425170 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:42:46.425449 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.422445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:45763 (local address 127.9.130.3:52977)
0928 02:42:46.422600 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:46.422608 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:46.422660 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:46.422786 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:46.422787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:46.422788 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:46.422929 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.422931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.423656 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.423656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.424451 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:46.424453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.424980 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.424981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:46.425037 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.425040 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:42:46.425111 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:42:46.425143 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":16,"thread_start_us":81,"threads_started":1}
I20260928 02:42:46.425611 29532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6402eca6ee034b1692306999ce5785fe" candidate_uuid: "b79cd424e129470888e06bdb4daa0af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56efd46f3eb44f2086f7480855959bce" is_pre_election: true
I20260928 02:42:46.425796 29532 raft_consensus.cc:2454] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79cd424e129470888e06bdb4daa0af2 in term 0.
I20260928 02:42:46.425828 29663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6402eca6ee034b1692306999ce5785fe" candidate_uuid: "b79cd424e129470888e06bdb4daa0af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb631c8eaaf04e749c05dbe933f56514" is_pre_election: true
I20260928 02:42:46.425998 29663 raft_consensus.cc:2454] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79cd424e129470888e06bdb4daa0af2 in term 0.
I20260928 02:42:46.426044 29730 leader_election.cc:302] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56efd46f3eb44f2086f7480855959bce, b79cd424e129470888e06bdb4daa0af2; no voters:
I20260928 02:42:46.426157 29862 raft_consensus.cc:2781] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:42:46.426211 29862 raft_consensus.cc:482] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:42:46.426272 29862 raft_consensus.cc:3037] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:46.426944 29862 raft_consensus.cc:504] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56efd46f3eb44f2086f7480855959bce" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 39617 } } peers { permanent_uuid: "b79cd424e129470888e06bdb4daa0af2" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43037 } } peers { permanent_uuid: "cb631c8eaaf04e749c05dbe933f56514" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45763 } }
I20260928 02:42:46.427076 29862 leader_election.cc:288] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [CANDIDATE]: Term 1 election: Requested vote from peers 56efd46f3eb44f2086f7480855959bce (127.9.130.1:39617), cb631c8eaaf04e749c05dbe933f56514 (127.9.130.2:45763)
I20260928 02:42:46.427213 29663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6402eca6ee034b1692306999ce5785fe" candidate_uuid: "b79cd424e129470888e06bdb4daa0af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb631c8eaaf04e749c05dbe933f56514"
I20260928 02:42:46.427276 29663 raft_consensus.cc:3037] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:46.427979 29663 raft_consensus.cc:2454] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79cd424e129470888e06bdb4daa0af2 in term 1.
I20260928 02:42:46.428089 29532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6402eca6ee034b1692306999ce5785fe" candidate_uuid: "b79cd424e129470888e06bdb4daa0af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56efd46f3eb44f2086f7480855959bce"
I20260928 02:42:46.428145 29729 leader_election.cc:302] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b79cd424e129470888e06bdb4daa0af2, cb631c8eaaf04e749c05dbe933f56514; no voters:
I20260928 02:42:46.428164 29532 raft_consensus.cc:3037] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:46.428232 29862 raft_consensus.cc:2781] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:46.428372 29862 raft_consensus.cc:686] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [term 1 LEADER]: Becoming Leader. State: Replica: b79cd424e129470888e06bdb4daa0af2, State: Running, Role: LEADER
I20260928 02:42:46.428478 29862 consensus_queue.cc:237] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56efd46f3eb44f2086f7480855959bce" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 39617 } } peers { permanent_uuid: "b79cd424e129470888e06bdb4daa0af2" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43037 } } peers { permanent_uuid: "cb631c8eaaf04e749c05dbe933f56514" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45763 } }
I20260928 02:42:46.429085 29532 raft_consensus.cc:2454] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79cd424e129470888e06bdb4daa0af2 in term 1.
I20260928 02:42:46.429227 29389 catalog_manager.cc:5495] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b79cd424e129470888e06bdb4daa0af2 (127.9.130.3). New cstate: current_term: 1 leader_uuid: "b79cd424e129470888e06bdb4daa0af2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56efd46f3eb44f2086f7480855959bce" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79cd424e129470888e06bdb4daa0af2" member_type: VOTER last_known_addr { host: "127.9.130.3" port: 43037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb631c8eaaf04e749c05dbe933f56514" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45763 } health_report { overall_health: UNKNOWN } } }
I20260928 02:42:46.436776 29853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.434450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.9.130.3:43037)
0928 02:42:46.434492 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:46.434494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:46.434503 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:42:46.434558 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:46.434559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:46.434574 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:46.434641 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:42:46.434760 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.435189 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.435989 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:46.436193 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:46.436293 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:46.436465 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:42:46.436558 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:42:46.436649 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:46.436672 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 02:42:46.436689 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:42:46.438446 29663 raft_consensus.cc:1261] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514 [term 1 FOLLOWER]: Refusing update from remote peer b79cd424e129470888e06bdb4daa0af2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:46.438508 29532 raft_consensus.cc:1261] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce [term 1 FOLLOWER]: Refusing update from remote peer b79cd424e129470888e06bdb4daa0af2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:46.438814 29864 consensus_queue.cc:1035] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56efd46f3eb44f2086f7480855959bce" member_type: VOTER last_known_addr { host: "127.9.130.1" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:46.438961 29862 consensus_queue.cc:1035] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb631c8eaaf04e749c05dbe933f56514" member_type: VOTER last_known_addr { host: "127.9.130.2" port: 45763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:46.441570 29868 mvcc.cc:205] Tried to move back new op lower bound from 7334147548929261568 to 7334147548891766784. Current Snapshot: MvccSnapshot[applied={T|T < 7334147548929261568}]
I20260928 02:42:46.441488 29867 mvcc.cc:205] Tried to move back new op lower bound from 7334147548929261568 to 7334147548891766784. Current Snapshot: MvccSnapshot[applied={T|T < 7334147548929261568}]
I20260928 02:42:46.443629 29389 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:46.443706 29389 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:46.444561 29389 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8c6c979a03e41f380e09807a34ad853: Sending DeleteTablet for 3 replicas of tablet 6402eca6ee034b1692306999ce5785fe
I20260928 02:42:46.444911 29643 tablet_service.cc:1514] Processing DeleteTablet for tablet 6402eca6ee034b1692306999ce5785fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:46 UTC) from {username='slave'} at 127.0.0.1:38814
I20260928 02:42:46.444973 29774 tablet_service.cc:1514] Processing DeleteTablet for tablet 6402eca6ee034b1692306999ce5785fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:46 UTC) from {username='slave'} at 127.0.0.1:49784
I20260928 02:42:46.445137 29512 tablet_service.cc:1514] Processing DeleteTablet for tablet 6402eca6ee034b1692306999ce5785fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:46 UTC) from {username='slave'} at 127.0.0.1:36452
I20260928 02:42:46.445194 29878 tablet_replica.cc:330] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2: stopping tablet replica
I20260928 02:42:46.445288 29878 raft_consensus.cc:2229] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:42:46.445385 29878 raft_consensus.cc:2258] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:46.445431 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 29450
I20260928 02:42:46.445399 29879 tablet_replica.cc:330] T 6402eca6ee034b1692306999ce5785fe P 56efd46f3eb44f2086f7480855959bce: stopping tablet replica
I20260928 02:42:46.446396 29878 ts_tablet_manager.cc:1905] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:46.448414 29878 ts_tablet_manager.cc:1918] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:46.448487 29878 log.cc:1196] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2: Deleting WAL directory at /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563127283700-9736-0/minicluster-data/ts-2/wal/wals/6402eca6ee034b1692306999ce5785fe
I20260928 02:42:46.448748 29878 ts_tablet_manager.cc:1939] T 6402eca6ee034b1692306999ce5785fe P b79cd424e129470888e06bdb4daa0af2: Deleting consensus metadata
I20260928 02:42:46.449288 29377 catalog_manager.cc:4841] TS b79cd424e129470888e06bdb4daa0af2 (127.9.130.3:43037): tablet 6402eca6ee034b1692306999ce5785fe (table test-table [id=6b2dbadf094c47a8aa101c05bb3253ea]) successfully deleted
W20260928 02:42:46.452000 29377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.130.1:39617 (error 108)
I20260928 02:42:46.452255 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 29581
I20260928 02:42:46.452289 29877 tablet_replica.cc:330] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514: stopping tablet replica
I20260928 02:42:46.452384 29877 raft_consensus.cc:2229] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:42:46.452481 29877 raft_consensus.cc:2258] T 6402eca6ee034b1692306999ce5785fe P cb631c8eaaf04e749c05dbe933f56514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:46.452550 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.452437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.1:39617 (local address 127.0.0.1:36468)
0928 02:42:46.452484 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:46.452518 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.1:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:42:46.452651 29377 catalog_manager.cc:4584] TS 56efd46f3eb44f2086f7480855959bce (127.9.130.1:39617): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6402eca6ee034b1692306999ce5785fe: Network error: Client connection negotiation failed: client connection to 127.9.130.1:39617: connect: Connection refused (error 111)
I20260928 02:42:46.457825 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 29712
I20260928 02:42:46.457859 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:46.457762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.2:45763 (local address 127.0.0.1:38816)
0928 02:42:46.457806 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:46.457836 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.2:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 02:42:46.462337 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 29359
2026-09-28T02:42:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (698 ms)
[----------] 6 tests from EncryptionPolicyTest (8155 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:46 dist-test-slave-kkrh krb5kdc[29883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:46 dist-test-slave-kkrh krb5kdc[29883](info): set up 2 sockets
Sep 28 02:42:46 dist-test-slave-kkrh krb5kdc[29883](info): commencing operation
krb5kdc: starting...
W20260928 02:42:47.495779 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:42:47 dist-test-slave-kkrh krb5kdc[29883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:42:47.610455 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:47.689366 29896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:47.689599 29896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:47.689635 29896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:47.690768 29896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:47.690833 29896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:47.690860 29896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:47.690891 29896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:47.690914 29896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:47.692399 29896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42693
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:47.692662 29896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:47.692878 29896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:47.697711 29901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:47.697724 29902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:47.697834 29904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:47.697912 29896 server_base.cc:1034] running on GCE node
I20260928 02:42:47.698143 29896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:47.698417 29896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:47.699581 29896 hybrid_clock.cc:648] HybridClock initialized: now 1790563367699578 us; error 39 us; skew 500 ppm
Sep 28 02:42:47 dist-test-slave-kkrh krb5kdc[29883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:47.706137 29896 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:42:47.706652 29896 webserver.cc:460] Webserver started at http://127.0.0.1:34639/ using document root <none> and password file <none>
I20260928 02:42:47.706887 29896 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:47.706955 29896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:47.707109 29896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:47.708256 29896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "b5ba37447a614b93bd481e890b3f92cd"
format_stamp: "Formatted at 2026-09-28 02:42:47 on dist-test-slave-kkrh"
server_key: "c9f200b602cabbdd4eb4aae7fea85769"
server_key_iv: "6d28ac29c4e2a28e969b99342b7147ac"
server_key_version: "encryptionkey@0"
I20260928 02:42:47.708624 29896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "b5ba37447a614b93bd481e890b3f92cd"
format_stamp: "Formatted at 2026-09-28 02:42:47 on dist-test-slave-kkrh"
server_key: "c9f200b602cabbdd4eb4aae7fea85769"
server_key_iv: "6d28ac29c4e2a28e969b99342b7147ac"
server_key_version: "encryptionkey@0"
I20260928 02:42:47.709920 29896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:42:47.710652 29911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:47.710819 29896 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:47.710922 29896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "b5ba37447a614b93bd481e890b3f92cd"
format_stamp: "Formatted at 2026-09-28 02:42:47 on dist-test-slave-kkrh"
server_key: "c9f200b602cabbdd4eb4aae7fea85769"
server_key_iv: "6d28ac29c4e2a28e969b99342b7147ac"
server_key_version: "encryptionkey@0"
I20260928 02:42:47.711001 29896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:47.743160 29896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:47.745368 29896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:47.745543 29896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:47.749607 29896 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42693
I20260928 02:42:47.749646 29963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42693 every 8 connection(s)
I20260928 02:42:47.750026 29896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:47.750970 29964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:47.753262 29964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd: Bootstrap starting.
I20260928 02:42:47.753851 29964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:47.754297 29964 log.cc:824] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:47.754951 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 29896
I20260928 02:42:47.755057 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:47.755160 29964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd: No bootstrap required, opened a new log
I20260928 02:42:47.755255 9736 external_mini_cluster.cc:1428] Setting key e3d82a9c28e091f7649e80cdd4827d43
I20260928 02:42:47.757063 29964 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ba37447a614b93bd481e890b3f92cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42693 } }
I20260928 02:42:47.757198 29964 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:47.757221 29964 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5ba37447a614b93bd481e890b3f92cd, State: Initialized, Role: FOLLOWER
I20260928 02:42:47.757372 29964 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ba37447a614b93bd481e890b3f92cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42693 } }
I20260928 02:42:47.757470 29964 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:47.757507 29964 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:47.757556 29964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:47.758369 29964 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ba37447a614b93bd481e890b3f92cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42693 } }
I20260928 02:42:47.758491 29964 leader_election.cc:302] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5ba37447a614b93bd481e890b3f92cd; no voters:
I20260928 02:42:47.758684 29964 leader_election.cc:288] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:47.758704 29969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:47.758809 29969 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [term 1 LEADER]: Becoming Leader. State: Replica: b5ba37447a614b93bd481e890b3f92cd, State: Running, Role: LEADER
I20260928 02:42:47.758944 29964 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:47.758957 29969 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ba37447a614b93bd481e890b3f92cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42693 } }
I20260928 02:42:47.759429 29971 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5ba37447a614b93bd481e890b3f92cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ba37447a614b93bd481e890b3f92cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42693 } } }
I20260928 02:42:47.759671 29971 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:47.759598 29970 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5ba37447a614b93bd481e890b3f92cd. Latest consensus state: current_term: 1 leader_uuid: "b5ba37447a614b93bd481e890b3f92cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ba37447a614b93bd481e890b3f92cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42693 } } }
I20260928 02:42:47.759982 29970 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd [sys.catalog]: This master's current role is: LEADER
W20260928 02:42:47.760810 29985 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:42:47.760867 29985 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:42:47.760919 29978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:47.761078 29978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:47.762612 29978 catalog_manager.cc:1347] Generated new cluster ID: beddfdcb953f470983acd0738f19264f
I20260928 02:42:47.762665 29978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:42:47 dist-test-slave-kkrh krb5kdc[29883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:42:47.767695 29968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:47.755805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56067 (local address 127.0.0.1:42693)
0928 02:42:47.755973 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:47.755981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:47.756006 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:42:47.757089 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:47.757097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:47.757151 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:47.757325 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:47.758098 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:47.759259 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:47.760329 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:47.760835 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:47.760983 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:47.764023 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:47.764026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:47.764026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:47.764036 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:47.765784 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:47.766192 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:47.766195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:47.766195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:47.766571 (+ 376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:47.766828 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:47.766834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:47.766834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:47.767068 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:47.767259 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:47.767352 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 02:42:47.767547 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260928 02:42:47.768657 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:42:47.778059 29968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:47.770004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58855 (local address 127.0.0.1:42693)
0928 02:42:47.770038 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:47.770040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:47.771994 (+ 1954us) server_negotiation.cc:407] Connection header received
0928 02:42:47.772072 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:47.772073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:47.772082 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:47.772137 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:47.772641 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:47.773153 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:47.773995 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:47.774212 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:47.774285 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:47.775441 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:47.775442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:47.775442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:47.775449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:47.776851 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:47.777229 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:47.777230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:47.777231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:47.777288 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:47.777507 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:47.777508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:47.777509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:47.777634 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:47.777743 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:47.777864 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 02:42:47.777934 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:42:47.781678 29978 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:47.782240 29978 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:47.787566 29978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5ba37447a614b93bd481e890b3f92cd: Generated new TSK 0
I20260928 02:42:47.787755 29978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:47.792037 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 29896
W20260928 02:42:47.796274 29881 connection.cc:535] client connection to 127.0.0.1:42693 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42693): Connection reset by peer (error 104)
2026-09-28T02:42:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:47 dist-test-slave-kkrh krb5kdc[29995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:47 dist-test-slave-kkrh krb5kdc[29995](info): set up 2 sockets
Sep 28 02:42:47 dist-test-slave-kkrh krb5kdc[29995](info): commencing operation
krb5kdc: starting...
W20260928 02:42:48.823909 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:42:48 dist-test-slave-kkrh krb5kdc[29995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:42:48.943055 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33029
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:49.022668 30008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:49.022912 30008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:49.022949 30008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:49.024116 30008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:49.024184 30008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:49.024209 30008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:49.024237 30008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:49.024278 30008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:49.025735 30008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33029
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:49.026014 30008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:49.026237 30008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:49.031246 30013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:49.031261 30014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:49.031327 30008 server_base.cc:1034] running on GCE node
W20260928 02:42:49.031261 30016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:49.031728 30008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:49.032029 30008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:49.033175 30008 hybrid_clock.cc:648] HybridClock initialized: now 1790563369033162 us; error 28 us; skew 500 ppm
Sep 28 02:42:49 dist-test-slave-kkrh krb5kdc[29995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:49.035558 30008 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:42:49.036059 30008 webserver.cc:460] Webserver started at http://127.0.0.1:46471/ using document root <none> and password file <none>
I20260928 02:42:49.036289 30008 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:49.036356 30008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:49.036500 30008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:49.037530 30008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "dcbaf5bd2fff4f499570d4bfc1575955"
format_stamp: "Formatted at 2026-09-28 02:42:49 on dist-test-slave-kkrh"
server_key: "7ef58177cf1cb5fb392d58c0fb139fce"
server_key_iv: "39157e11043a0aa5c67c944654f7f817"
server_key_version: "encryptionkey@0"
I20260928 02:42:49.037899 30008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "dcbaf5bd2fff4f499570d4bfc1575955"
format_stamp: "Formatted at 2026-09-28 02:42:49 on dist-test-slave-kkrh"
server_key: "7ef58177cf1cb5fb392d58c0fb139fce"
server_key_iv: "39157e11043a0aa5c67c944654f7f817"
server_key_version: "encryptionkey@0"
I20260928 02:42:49.039180 30008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:42:49.039917 30023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:49.040083 30008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:49.040189 30008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "dcbaf5bd2fff4f499570d4bfc1575955"
format_stamp: "Formatted at 2026-09-28 02:42:49 on dist-test-slave-kkrh"
server_key: "7ef58177cf1cb5fb392d58c0fb139fce"
server_key_iv: "39157e11043a0aa5c67c944654f7f817"
server_key_version: "encryptionkey@0"
I20260928 02:42:49.040284 30008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:49.071174 30008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:49.073118 30008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:49.073292 30008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:49.077250 30008 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33029
I20260928 02:42:49.077279 30075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33029 every 8 connection(s)
I20260928 02:42:49.077633 30008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:49.077811 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 30008
I20260928 02:42:49.077934 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:49.078157 9736 external_mini_cluster.cc:1428] Setting key 54dfab5de5369fd1130772ead139b5e4
I20260928 02:42:49.078893 30076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:49.081831 30076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955: Bootstrap starting.
I20260928 02:42:49.082442 30076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:49.082758 30076 log.cc:824] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:49.083491 30076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955: No bootstrap required, opened a new log
I20260928 02:42:49.083793 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:49.078708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37281 (local address 127.0.0.1:33029)
0928 02:42:49.078896 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:49.078905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:49.078961 (+ 56us) server_negotiation.cc:407] Connection header received
0928 02:42:49.079415 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:49.079422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:49.079474 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:49.079586 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:49.079886 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:49.080887 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:49.081972 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:49.082378 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:49.082459 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:49.083218 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:49.083223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:49.083223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:49.083385 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:49.083413 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:49.083424 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:49.083517 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:49.083598 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 02:42:49.083627 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":105,"threads_started":1}
I20260928 02:42:49.084859 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:42:49.085870 30076 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbaf5bd2fff4f499570d4bfc1575955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33029 } }
I20260928 02:42:49.086030 30076 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:49.086076 30076 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbaf5bd2fff4f499570d4bfc1575955, State: Initialized, Role: FOLLOWER
I20260928 02:42:49.086196 30076 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbaf5bd2fff4f499570d4bfc1575955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33029 } }
I20260928 02:42:49.086290 30076 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:49.086313 30076 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:49.086386 30076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:49.087049 30076 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbaf5bd2fff4f499570d4bfc1575955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33029 } }
I20260928 02:42:49.087152 30076 leader_election.cc:302] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcbaf5bd2fff4f499570d4bfc1575955; no voters:
I20260928 02:42:49.087320 30076 leader_election.cc:288] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:49.087376 30086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:49.087529 30086 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [term 1 LEADER]: Becoming Leader. State: Replica: dcbaf5bd2fff4f499570d4bfc1575955, State: Running, Role: LEADER
I20260928 02:42:49.087611 30076 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:49.087668 30086 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbaf5bd2fff4f499570d4bfc1575955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33029 } }
I20260928 02:42:49.088055 30086 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcbaf5bd2fff4f499570d4bfc1575955. Latest consensus state: current_term: 1 leader_uuid: "dcbaf5bd2fff4f499570d4bfc1575955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbaf5bd2fff4f499570d4bfc1575955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33029 } } }
I20260928 02:42:49.088254 30086 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:49.088187 30088 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcbaf5bd2fff4f499570d4bfc1575955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbaf5bd2fff4f499570d4bfc1575955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33029 } } }
I20260928 02:42:49.088486 30088 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:49.088779 30095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:49.089325 30095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:49.089833 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:49.086180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48335 (local address 127.0.0.1:33029)
0928 02:42:49.086226 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:49.086228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:49.086353 (+ 125us) server_negotiation.cc:407] Connection header received
0928 02:42:49.086405 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:49.086405 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:49.086415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:49.086465 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:49.086622 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:49.087300 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:49.088385 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:49.088669 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:49.088728 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:49.089489 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:49.089490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:49.089491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:49.089495 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:49.089507 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:49.089511 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:49.089542 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:49.089753 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 02:42:49.089764 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:42:49.090786 30095 catalog_manager.cc:1347] Generated new cluster ID: 3c7c729e5f9b4887a578cf4514d4ca2f
I20260928 02:42:49.090842 30095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:49.102998 30095 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:49.103685 30095 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:49.110778 30095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcbaf5bd2fff4f499570d4bfc1575955: Generated new TSK 0
I20260928 02:42:49.110942 30095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:49.115358 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 30008
W20260928 02:42:49.119889 29993 connection.cc:535] client connection to 127.0.0.1:33029 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33029): Connection reset by peer (error 104)
2026-09-28T02:42:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:49 dist-test-slave-kkrh krb5kdc[30107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:49 dist-test-slave-kkrh krb5kdc[30107](info): set up 2 sockets
Sep 28 02:42:49 dist-test-slave-kkrh krb5kdc[30107](info): commencing operation
krb5kdc: starting...
W20260928 02:42:50.147739 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:42:50 dist-test-slave-kkrh krb5kdc[30107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:42:50.260308 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37447
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:50.338994 30120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:50.339218 30120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:50.339254 30120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:50.340407 30120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:50.340476 30120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:50.340507 30120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:50.340539 30120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:50.340561 30120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:50.342013 30120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37447
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:50.342288 30120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:50.342519 30120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:50.347637 30128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:50.347662 30125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:50.347748 30120 server_base.cc:1034] running on GCE node
W20260928 02:42:50.347637 30126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:50.348116 30120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:50.348438 30120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:50.349603 30120 hybrid_clock.cc:648] HybridClock initialized: now 1790563370349604 us; error 58 us; skew 500 ppm
Sep 28 02:42:50 dist-test-slave-kkrh krb5kdc[30107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:50.352083 30120 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:42:50.352551 30120 webserver.cc:460] Webserver started at http://127.0.0.1:39903/ using document root <none> and password file <none>
I20260928 02:42:50.352789 30120 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:50.352855 30120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:50.353008 30120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:50.354069 30120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "d96bf0aa62b8443c913842ed33562211"
format_stamp: "Formatted at 2026-09-28 02:42:50 on dist-test-slave-kkrh"
server_key: "1b3698094759972591971272ad40332e"
server_key_iv: "23934d081ef8d4a0a000228a7152301d"
server_key_version: "encryptionkey@0"
I20260928 02:42:50.354449 30120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "d96bf0aa62b8443c913842ed33562211"
format_stamp: "Formatted at 2026-09-28 02:42:50 on dist-test-slave-kkrh"
server_key: "1b3698094759972591971272ad40332e"
server_key_iv: "23934d081ef8d4a0a000228a7152301d"
server_key_version: "encryptionkey@0"
I20260928 02:42:50.355867 30120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:42:50.356634 30135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:50.356796 30120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:50.356909 30120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "d96bf0aa62b8443c913842ed33562211"
format_stamp: "Formatted at 2026-09-28 02:42:50 on dist-test-slave-kkrh"
server_key: "1b3698094759972591971272ad40332e"
server_key_iv: "23934d081ef8d4a0a000228a7152301d"
server_key_version: "encryptionkey@0"
I20260928 02:42:50.356998 30120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:50.360208 30120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:50.362006 30120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:50.362155 30120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:50.366057 30120 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37447
I20260928 02:42:50.366089 30187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37447 every 8 connection(s)
I20260928 02:42:50.366405 30120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:50.367350 30188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:50.369551 30188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211: Bootstrap starting.
I20260928 02:42:50.370090 30188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:50.370368 30188 log.cc:824] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:50.371029 30188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211: No bootstrap required, opened a new log
I20260928 02:42:50.372660 30188 raft_consensus.cc:348] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96bf0aa62b8443c913842ed33562211" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37447 } }
I20260928 02:42:50.372789 30188 raft_consensus.cc:374] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:50.372810 30188 raft_consensus.cc:729] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d96bf0aa62b8443c913842ed33562211, State: Initialized, Role: FOLLOWER
I20260928 02:42:50.372956 30188 consensus_queue.cc:260] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96bf0aa62b8443c913842ed33562211" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37447 } }
I20260928 02:42:50.373020 30188 raft_consensus.cc:388] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:50.373040 30188 raft_consensus.cc:482] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:50.373121 30188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:50.373644 30188 raft_consensus.cc:504] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96bf0aa62b8443c913842ed33562211" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37447 } }
I20260928 02:42:50.373728 30188 leader_election.cc:302] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d96bf0aa62b8443c913842ed33562211; no voters:
I20260928 02:42:50.373894 30188 leader_election.cc:288] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:50.373951 30191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:50.374146 30188 sys_catalog.cc:564] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:50.374171 30191 raft_consensus.cc:686] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [term 1 LEADER]: Becoming Leader. State: Replica: d96bf0aa62b8443c913842ed33562211, State: Running, Role: LEADER
I20260928 02:42:50.374272 30191 consensus_queue.cc:237] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96bf0aa62b8443c913842ed33562211" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37447 } }
I20260928 02:42:50.374716 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 30120
I20260928 02:42:50.374850 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:50.375042 9736 external_mini_cluster.cc:1428] Setting key 311cb2236d73bd0fbbbd3858876a1904
I20260928 02:42:50.375862 30192 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d96bf0aa62b8443c913842ed33562211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96bf0aa62b8443c913842ed33562211" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37447 } } }
I20260928 02:42:50.375888 30193 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d96bf0aa62b8443c913842ed33562211. Latest consensus state: current_term: 1 leader_uuid: "d96bf0aa62b8443c913842ed33562211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96bf0aa62b8443c913842ed33562211" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37447 } } }
I20260928 02:42:50.375954 30192 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:50.375972 30193 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:50.376156 30208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:50.376730 30208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:50.377543 30207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:50.375775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34955 (local address 127.0.0.1:37447)
0928 02:42:50.375960 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:50.375969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:50.376004 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:42:50.376134 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:50.376142 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:50.376172 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:50.376326 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:50.377091 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:50.377095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:50.377095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:50.377099 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:50.377140 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:50.377154 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:50.377228 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:50.377321 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 02:42:50.377360 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":106,"threads_started":1}
I20260928 02:42:50.378358 30208 catalog_manager.cc:1347] Generated new cluster ID: 0c5639767e804baea6b20a632783ee76
I20260928 02:42:50.378407 30208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:50.378432 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:42:50.384770 30207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:50.379909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43269 (local address 127.0.0.1:37447)
0928 02:42:50.379941 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:50.379942 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:50.383737 (+ 3795us) server_negotiation.cc:407] Connection header received
0928 02:42:50.383798 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:50.383799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:50.383804 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:50.383875 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:50.384475 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:50.384476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:50.384476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:50.384480 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:50.384494 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:50.384498 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:50.384557 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:50.384621 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 02:42:50.384634 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:42:50.390210 30208 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:50.390357 30208 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:50.394904 30208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d96bf0aa62b8443c913842ed33562211: Generated new TSK 0
I20260928 02:42:50.395083 30208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:50.401932 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 30120
2026-09-28T02:42:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:50 dist-test-slave-kkrh krb5kdc[30219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:50 dist-test-slave-kkrh krb5kdc[30219](info): set up 2 sockets
Sep 28 02:42:50 dist-test-slave-kkrh krb5kdc[30219](info): commencing operation
krb5kdc: starting...
W20260928 02:42:51.431870 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:42:51 dist-test-slave-kkrh krb5kdc[30219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:42:51.547495 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38291
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:51.627481 30232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:51.627727 30232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:51.627764 30232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:51.628943 30232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:51.629017 30232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:51.629041 30232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:51.629060 30232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:51.629117 30232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:51.630602 30232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:51.630879 30232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:51.631102 30232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:51.636026 30238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:51.636044 30237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:51.636085 30232 server_base.cc:1034] running on GCE node
W20260928 02:42:51.636044 30240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:51.636448 30232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:51.636683 30232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:51.637822 30232 hybrid_clock.cc:648] HybridClock initialized: now 1790563371637820 us; error 37 us; skew 500 ppm
Sep 28 02:42:51 dist-test-slave-kkrh krb5kdc[30219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:51.640237 30232 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:42:51.640695 30232 webserver.cc:460] Webserver started at http://127.0.0.1:40885/ using document root <none> and password file <none>
I20260928 02:42:51.640916 30232 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:51.640982 30232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:51.641149 30232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:51.642174 30232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "25e6a2e5b96e4560ad1931b59e4af6b3"
format_stamp: "Formatted at 2026-09-28 02:42:51 on dist-test-slave-kkrh"
server_key: "3037f1e8a974f9275335a5d8f65fdd0d"
server_key_iv: "21aff28490504e94fb22fb9de125fb71"
server_key_version: "encryptionkey@0"
I20260928 02:42:51.642534 30232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "25e6a2e5b96e4560ad1931b59e4af6b3"
format_stamp: "Formatted at 2026-09-28 02:42:51 on dist-test-slave-kkrh"
server_key: "3037f1e8a974f9275335a5d8f65fdd0d"
server_key_iv: "21aff28490504e94fb22fb9de125fb71"
server_key_version: "encryptionkey@0"
I20260928 02:42:51.643805 30232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:42:51.644644 30247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:51.644824 30232 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:42:51.644934 30232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "25e6a2e5b96e4560ad1931b59e4af6b3"
format_stamp: "Formatted at 2026-09-28 02:42:51 on dist-test-slave-kkrh"
server_key: "3037f1e8a974f9275335a5d8f65fdd0d"
server_key_iv: "21aff28490504e94fb22fb9de125fb71"
server_key_version: "encryptionkey@0"
I20260928 02:42:51.645022 30232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:51.659386 30232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:51.661430 30232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:51.661597 30232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:51.665642 30232 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38291
I20260928 02:42:51.665686 30299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38291 every 8 connection(s)
I20260928 02:42:51.666043 30232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:51.667047 30300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:51.669457 30300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3: Bootstrap starting.
I20260928 02:42:51.670023 30300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:51.670275 30300 log.cc:824] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:51.670945 30300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3: No bootstrap required, opened a new log
I20260928 02:42:51.671988 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 30232
I20260928 02:42:51.672128 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:51.672341 9736 external_mini_cluster.cc:1428] Setting key 1a1ddbc2835ed30d791f8ff2dc75f727
I20260928 02:42:51.673310 30300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e6a2e5b96e4560ad1931b59e4af6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38291 } }
I20260928 02:42:51.673498 30300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:51.673555 30300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e6a2e5b96e4560ad1931b59e4af6b3, State: Initialized, Role: FOLLOWER
I20260928 02:42:51.673710 30300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e6a2e5b96e4560ad1931b59e4af6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38291 } }
I20260928 02:42:51.673808 30300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:51.673847 30300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:51.673905 30300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:51.674969 30300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e6a2e5b96e4560ad1931b59e4af6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38291 } }
I20260928 02:42:51.675097 30300 leader_election.cc:302] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25e6a2e5b96e4560ad1931b59e4af6b3; no voters:
I20260928 02:42:51.675287 30300 leader_election.cc:288] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:51.675334 30305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:51.675504 30305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [term 1 LEADER]: Becoming Leader. State: Replica: 25e6a2e5b96e4560ad1931b59e4af6b3, State: Running, Role: LEADER
I20260928 02:42:51.675554 30300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:51.675697 30305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e6a2e5b96e4560ad1931b59e4af6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38291 } }
I20260928 02:42:51.676139 30306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25e6a2e5b96e4560ad1931b59e4af6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e6a2e5b96e4560ad1931b59e4af6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38291 } } }
I20260928 02:42:51.676218 30306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:51.676383 30307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25e6a2e5b96e4560ad1931b59e4af6b3. Latest consensus state: current_term: 1 leader_uuid: "25e6a2e5b96e4560ad1931b59e4af6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e6a2e5b96e4560ad1931b59e4af6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38291 } } }
I20260928 02:42:51.676427 30307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:51.676898 30311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:51.677423 30311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:51.679240 30311 catalog_manager.cc:1347] Generated new cluster ID: 3b77a4fb155f4597bda70f60dcdf0b27
I20260928 02:42:51.679296 30311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:42:51 dist-test-slave-kkrh krb5kdc[30219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:42:51.685032 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:51.672880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56819 (local address 127.0.0.1:38291)
0928 02:42:51.673043 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:51.673058 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:51.673082 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:42:51.673291 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:51.673296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:51.673338 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:51.673450 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:51.674367 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:51.675612 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:51.676859 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:51.677527 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:51.678037 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:51.680803 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:51.680807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:51.680808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:51.680821 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:51.682885 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:51.683333 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:51.683336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:51.683336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:51.683464 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:51.683702 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:51.683703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:51.683704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:51.684013 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:51.684614 (+ 601us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:51.684696 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 02:42:51.684834 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260928 02:42:51.686378 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:42:51.703770 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:51.688063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54755 (local address 127.0.0.1:38291)
0928 02:42:51.688101 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:51.688103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:51.696024 (+ 7921us) server_negotiation.cc:407] Connection header received
0928 02:42:51.696093 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:51.696094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:51.696106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:51.696167 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:51.696849 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:51.697637 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:51.698779 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:51.699126 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:51.699191 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:51.700815 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:51.700816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:51.700817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:51.700824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:51.702547 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:51.702962 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:51.702963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:51.702963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:51.703022 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:51.703241 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:51.703241 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:51.703242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:51.703362 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:51.703465 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:51.703592 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:42:51.703665 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:42:51.726704 30311 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:51.727525 30311 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:51.737754 30311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25e6a2e5b96e4560ad1931b59e4af6b3: Generated new TSK 0
I20260928 02:42:51.737968 30311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:51.747339 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 30232
2026-09-28T02:42:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:51 dist-test-slave-kkrh krb5kdc[30331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:51 dist-test-slave-kkrh krb5kdc[30331](info): set up 2 sockets
Sep 28 02:42:51 dist-test-slave-kkrh krb5kdc[30331](info): commencing operation
krb5kdc: starting...
W20260928 02:42:52.855986 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:42:52 dist-test-slave-kkrh krb5kdc[30331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:42:52.971385 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:53.052608 30344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:53.052881 30344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:53.052922 30344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:53.054090 30344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:53.054165 30344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:53.054193 30344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:53.054224 30344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:53.054258 30344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:53.055722 30344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:53.056033 30344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:53.056268 30344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:53.061242 30352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:53.061338 30350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:53.061282 30349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:53.061447 30344 server_base.cc:1034] running on GCE node
I20260928 02:42:53.061690 30344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:53.062001 30344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:53.063144 30344 hybrid_clock.cc:648] HybridClock initialized: now 1790563373063117 us; error 38 us; skew 500 ppm
Sep 28 02:42:53 dist-test-slave-kkrh krb5kdc[30331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:53.070667 30344 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:42:53.071122 30344 webserver.cc:460] Webserver started at http://127.0.0.1:40391/ using document root <none> and password file <none>
I20260928 02:42:53.071357 30344 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:53.071431 30344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:53.071565 30344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:53.072678 30344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "26ce23339f554c41ad9d1b04bdfd959f"
format_stamp: "Formatted at 2026-09-28 02:42:53 on dist-test-slave-kkrh"
server_key: "f13f9e870e197f881d43bec11c3898d6"
server_key_iv: "2caa5c1077daf1cdfde37ad5c462d95d"
server_key_version: "encryptionkey@0"
I20260928 02:42:53.073045 30344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "26ce23339f554c41ad9d1b04bdfd959f"
format_stamp: "Formatted at 2026-09-28 02:42:53 on dist-test-slave-kkrh"
server_key: "f13f9e870e197f881d43bec11c3898d6"
server_key_iv: "2caa5c1077daf1cdfde37ad5c462d95d"
server_key_version: "encryptionkey@0"
I20260928 02:42:53.074334 30344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:42:53.075079 30359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:53.075286 30344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:42:53.075392 30344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "26ce23339f554c41ad9d1b04bdfd959f"
format_stamp: "Formatted at 2026-09-28 02:42:53 on dist-test-slave-kkrh"
server_key: "f13f9e870e197f881d43bec11c3898d6"
server_key_iv: "2caa5c1077daf1cdfde37ad5c462d95d"
server_key_version: "encryptionkey@0"
I20260928 02:42:53.075493 30344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:53.098131 30344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:53.100430 30344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:53.100593 30344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:53.104619 30344 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43177
I20260928 02:42:53.104655 30411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43177 every 8 connection(s)
I20260928 02:42:53.105051 30344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:53.105868 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 30344
I20260928 02:42:53.105989 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:53.106132 30412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:53.106216 9736 external_mini_cluster.cc:1428] Setting key db15b4ad243355a2376994eb3612b2fc
I20260928 02:42:53.108865 30412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f: Bootstrap starting.
I20260928 02:42:53.109444 30412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:53.109771 30412 log.cc:824] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:53.110463 30412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f: No bootstrap required, opened a new log
I20260928 02:42:53.111640 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:53.106741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44605 (local address 127.0.0.1:43177)
0928 02:42:53.106963 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:53.106971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:53.106995 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:42:53.107469 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:53.107476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:53.107523 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:53.107670 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:53.107899 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:53.108775 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:53.109626 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:53.110195 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:53.110310 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:53.110955 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:53.110959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:53.110959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:53.111156 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:53.111194 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:53.111208 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:53.111282 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:53.111368 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 02:42:53.111430 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":91,"threads_started":1}
I20260928 02:42:53.112815 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:42:53.112880 30412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ce23339f554c41ad9d1b04bdfd959f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43177 } }
I20260928 02:42:53.113029 30412 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:53.113057 30412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ce23339f554c41ad9d1b04bdfd959f, State: Initialized, Role: FOLLOWER
I20260928 02:42:53.113173 30412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ce23339f554c41ad9d1b04bdfd959f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43177 } }
I20260928 02:42:53.113237 30412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:53.113270 30412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:53.113319 30412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:53.114003 30412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ce23339f554c41ad9d1b04bdfd959f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43177 } }
I20260928 02:42:53.114121 30412 leader_election.cc:302] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26ce23339f554c41ad9d1b04bdfd959f; no voters:
I20260928 02:42:53.114291 30412 leader_election.cc:288] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:53.114367 30421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:53.114521 30421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [term 1 LEADER]: Becoming Leader. State: Replica: 26ce23339f554c41ad9d1b04bdfd959f, State: Running, Role: LEADER
I20260928 02:42:53.114584 30412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:53.114645 30421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ce23339f554c41ad9d1b04bdfd959f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43177 } }
I20260928 02:42:53.114929 30421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26ce23339f554c41ad9d1b04bdfd959f. Latest consensus state: current_term: 1 leader_uuid: "26ce23339f554c41ad9d1b04bdfd959f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ce23339f554c41ad9d1b04bdfd959f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43177 } } }
I20260928 02:42:53.114946 30424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26ce23339f554c41ad9d1b04bdfd959f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ce23339f554c41ad9d1b04bdfd959f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43177 } } }
I20260928 02:42:53.115091 30421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:53.115099 30424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:53.115315 30429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:53.116030 30429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:53.117653 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:53.114295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39615 (local address 127.0.0.1:43177)
0928 02:42:53.114330 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:53.114331 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:53.114507 (+ 176us) server_negotiation.cc:407] Connection header received
0928 02:42:53.114533 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:53.114534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:53.114544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:53.114599 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:53.114795 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:53.115537 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:53.116563 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:53.116845 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:53.116928 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:53.117461 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:53.117462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:53.117462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:53.117465 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:53.117476 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:53.117479 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:53.117518 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:53.117564 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 02:42:53.117582 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:42:53.124013 30429 catalog_manager.cc:1347] Generated new cluster ID: 018607f7152d47bcb6225b5ea023bf4e
I20260928 02:42:53.124083 30429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:53.134893 30429 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:53.135505 30429 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:53.141667 30429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26ce23339f554c41ad9d1b04bdfd959f: Generated new TSK 0
I20260928 02:42:53.141891 30429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:53.151409 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 30344
2026-09-28T02:42:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:53 dist-test-slave-kkrh krb5kdc[30443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:53 dist-test-slave-kkrh krb5kdc[30443](info): set up 2 sockets
Sep 28 02:42:53 dist-test-slave-kkrh krb5kdc[30443](info): commencing operation
krb5kdc: starting...
W20260928 02:42:54.179947 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:42:54 dist-test-slave-kkrh krb5kdc[30443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:42:54.293838 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:54.372303 30456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:54.372525 30456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:54.372548 30456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:54.373690 30456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:54.373732 30456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:54.373746 30456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:54.373785 30456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:54.373814 30456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:54.375254 30456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:54.375497 30456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:54.375730 30456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:54.380486 30462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:54.380486 30464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:54.380525 30461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:54.380755 30456 server_base.cc:1034] running on GCE node
I20260928 02:42:54.380986 30456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:54.381219 30456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:54.382357 30456 hybrid_clock.cc:648] HybridClock initialized: now 1790563374382336 us; error 37 us; skew 500 ppm
Sep 28 02:42:54 dist-test-slave-kkrh krb5kdc[30443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:54.384859 30456 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:42:54.385300 30456 webserver.cc:460] Webserver started at http://127.0.0.1:38245/ using document root <none> and password file <none>
I20260928 02:42:54.385527 30456 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:54.385571 30456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:54.385725 30456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:54.386706 30456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "b83be1db379d4b97aa18ce0f4aa431bd"
format_stamp: "Formatted at 2026-09-28 02:42:54 on dist-test-slave-kkrh"
server_key: "257fd3f7a6b45f5219d85c5e4a784bee"
server_key_iv: "35493a82ff01b839b3a14843bd071278"
server_key_version: "encryptionkey@0"
I20260928 02:42:54.387059 30456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "b83be1db379d4b97aa18ce0f4aa431bd"
format_stamp: "Formatted at 2026-09-28 02:42:54 on dist-test-slave-kkrh"
server_key: "257fd3f7a6b45f5219d85c5e4a784bee"
server_key_iv: "35493a82ff01b839b3a14843bd071278"
server_key_version: "encryptionkey@0"
I20260928 02:42:54.388437 30456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:42:54.389158 30471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:54.389338 30456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:54.389410 30456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "b83be1db379d4b97aa18ce0f4aa431bd"
format_stamp: "Formatted at 2026-09-28 02:42:54 on dist-test-slave-kkrh"
server_key: "257fd3f7a6b45f5219d85c5e4a784bee"
server_key_iv: "35493a82ff01b839b3a14843bd071278"
server_key_version: "encryptionkey@0"
I20260928 02:42:54.389494 30456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:54.392817 30456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:54.395023 30456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:54.395169 30456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:54.399115 30456 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33195
I20260928 02:42:54.399143 30523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33195 every 8 connection(s)
I20260928 02:42:54.399466 30456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:54.400439 30524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:54.402534 30524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd: Bootstrap starting.
I20260928 02:42:54.403074 30524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:54.403352 30524 log.cc:824] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:54.404049 30524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd: No bootstrap required, opened a new log
I20260928 02:42:54.405838 30524 raft_consensus.cc:348] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83be1db379d4b97aa18ce0f4aa431bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } }
I20260928 02:42:54.405972 30524 raft_consensus.cc:374] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:54.405993 30524 raft_consensus.cc:729] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83be1db379d4b97aa18ce0f4aa431bd, State: Initialized, Role: FOLLOWER
I20260928 02:42:54.406124 30524 consensus_queue.cc:260] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83be1db379d4b97aa18ce0f4aa431bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } }
I20260928 02:42:54.406203 30524 raft_consensus.cc:388] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:54.406225 30524 raft_consensus.cc:482] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:54.406262 30524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:54.406737 30524 raft_consensus.cc:504] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83be1db379d4b97aa18ce0f4aa431bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } }
I20260928 02:42:54.406819 30524 leader_election.cc:302] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b83be1db379d4b97aa18ce0f4aa431bd; no voters:
I20260928 02:42:54.406939 30524 leader_election.cc:288] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:54.407002 30527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:54.407183 30524 sys_catalog.cc:564] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:54.407169 30527 raft_consensus.cc:686] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [term 1 LEADER]: Becoming Leader. State: Replica: b83be1db379d4b97aa18ce0f4aa431bd, State: Running, Role: LEADER
I20260928 02:42:54.407402 30527 consensus_queue.cc:237] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83be1db379d4b97aa18ce0f4aa431bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } }
I20260928 02:42:54.407981 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 30456
I20260928 02:42:54.408118 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:54.408262 30528 sys_catalog.cc:455] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b83be1db379d4b97aa18ce0f4aa431bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83be1db379d4b97aa18ce0f4aa431bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } } }
I20260928 02:42:54.408316 9736 external_mini_cluster.cc:1428] Setting key 0f55f9dd8c9e757833f27674605261c4
I20260928 02:42:54.408339 30528 sys_catalog.cc:458] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:54.408551 30529 sys_catalog.cc:455] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b83be1db379d4b97aa18ce0f4aa431bd. Latest consensus state: current_term: 1 leader_uuid: "b83be1db379d4b97aa18ce0f4aa431bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83be1db379d4b97aa18ce0f4aa431bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } } }
I20260928 02:42:54.408608 30529 sys_catalog.cc:458] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd [sys.catalog]: This master's current role is: LEADER
W20260928 02:42:54.409348 30543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:42:54.409408 30543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:42:54.409457 30539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:54.409614 30539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:54.410995 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:54.408959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58809 (local address 127.0.0.1:33195)
0928 02:42:54.409232 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:54.409242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:54.409274 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:42:54.409396 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:54.409403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:54.409432 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:54.409567 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:54.410535 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:54.410538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:54.410538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:54.410542 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:54.410580 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:54.410590 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:54.410673 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:54.410758 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:42:54.410809 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":69,"threads_started":1}
I20260928 02:42:54.411441 30539 catalog_manager.cc:1347] Generated new cluster ID: dc2289f6150c48968ebb23664bb0f30d
I20260928 02:42:54.411489 30539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:54.411942 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:42:54.414374 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:54.413427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49683 (local address 127.0.0.1:33195)
0928 02:42:54.413462 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:54.413464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:54.413475 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:42:54.413547 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:54.413548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:54.413553 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:54.413604 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:54.414149 (+ 545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:54.414151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:54.414151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:54.414155 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:54.414170 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:54.414175 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:54.414238 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:54.414293 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 02:42:54.414306 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:42:54.422395 30539 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:54.422564 30539 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:54.427731 30539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b83be1db379d4b97aa18ce0f4aa431bd: Generated new TSK 0
I20260928 02:42:54.427989 30539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:54.429843 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 30456
2026-09-28T02:42:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:54 dist-test-slave-kkrh krb5kdc[30555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:54 dist-test-slave-kkrh krb5kdc[30555](info): set up 2 sockets
Sep 28 02:42:54 dist-test-slave-kkrh krb5kdc[30555](info): commencing operation
krb5kdc: starting...
W20260928 02:42:55.456331 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:42:55 dist-test-slave-kkrh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:42:55.572517 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:55.650908 30568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:55.651101 30568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:55.651124 30568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:55.652302 30568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:55.652341 30568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:55.652356 30568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:55.652369 30568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:55.652381 30568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:55.653849 30568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:55.654085 30568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:55.654325 30568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:55.659147 30574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:55.659247 30576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:55.659154 30573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:55.659401 30568 server_base.cc:1034] running on GCE node
I20260928 02:42:55.659617 30568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:55.659911 30568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:55.661056 30568 hybrid_clock.cc:648] HybridClock initialized: now 1790563375661040 us; error 33 us; skew 500 ppm
Sep 28 02:42:55 dist-test-slave-kkrh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:55.663530 30568 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:42:55.664000 30568 webserver.cc:460] Webserver started at http://127.0.0.1:39605/ using document root <none> and password file <none>
I20260928 02:42:55.664223 30568 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:55.664268 30568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:55.664407 30568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:55.665426 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "686aaf6c350b447fbddec2aaaaa077c2"
format_stamp: "Formatted at 2026-09-28 02:42:55 on dist-test-slave-kkrh"
server_key: "e402a8677ee8776ef4184696a16df08e"
server_key_iv: "fe1899b8ab0f623d3aca45b97816bbfd"
server_key_version: "encryptionkey@0"
I20260928 02:42:55.665776 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "686aaf6c350b447fbddec2aaaaa077c2"
format_stamp: "Formatted at 2026-09-28 02:42:55 on dist-test-slave-kkrh"
server_key: "e402a8677ee8776ef4184696a16df08e"
server_key_iv: "fe1899b8ab0f623d3aca45b97816bbfd"
server_key_version: "encryptionkey@0"
I20260928 02:42:55.667044 30568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:42:55.667753 30583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:55.667940 30568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:42:55.668040 30568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "686aaf6c350b447fbddec2aaaaa077c2"
format_stamp: "Formatted at 2026-09-28 02:42:55 on dist-test-slave-kkrh"
server_key: "e402a8677ee8776ef4184696a16df08e"
server_key_iv: "fe1899b8ab0f623d3aca45b97816bbfd"
server_key_version: "encryptionkey@0"
I20260928 02:42:55.668115 30568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:55.685125 30568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:55.686843 30568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:55.687002 30568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:55.691012 30568 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42899
I20260928 02:42:55.691049 30635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42899 every 8 connection(s)
I20260928 02:42:55.691416 30568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:55.692481 30636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:55.695019 30636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2: Bootstrap starting.
I20260928 02:42:55.695636 30636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:55.695992 30636 log.cc:824] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:55.696753 30636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2: No bootstrap required, opened a new log
I20260928 02:42:55.696836 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 30568
I20260928 02:42:55.696985 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:55.697188 9736 external_mini_cluster.cc:1428] Setting key ce28824d54c25d44de326cbc8b47daa4
I20260928 02:42:55.698851 30636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "686aaf6c350b447fbddec2aaaaa077c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260928 02:42:55.699023 30636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:55.699065 30636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 686aaf6c350b447fbddec2aaaaa077c2, State: Initialized, Role: FOLLOWER
I20260928 02:42:55.699206 30636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "686aaf6c350b447fbddec2aaaaa077c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260928 02:42:55.699298 30636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:55.699340 30636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:55.699397 30636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:55.700495 30636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "686aaf6c350b447fbddec2aaaaa077c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260928 02:42:55.700632 30636 leader_election.cc:302] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 686aaf6c350b447fbddec2aaaaa077c2; no voters:
I20260928 02:42:55.700820 30636 leader_election.cc:288] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:55.700868 30641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:55.701021 30641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [term 1 LEADER]: Becoming Leader. State: Replica: 686aaf6c350b447fbddec2aaaaa077c2, State: Running, Role: LEADER
I20260928 02:42:55.701097 30636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:55.701114 30641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "686aaf6c350b447fbddec2aaaaa077c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260928 02:42:55.701428 30641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 686aaf6c350b447fbddec2aaaaa077c2. Latest consensus state: current_term: 1 leader_uuid: "686aaf6c350b447fbddec2aaaaa077c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "686aaf6c350b447fbddec2aaaaa077c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } } }
I20260928 02:42:55.701509 30641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:55.701438 30642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "686aaf6c350b447fbddec2aaaaa077c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "686aaf6c350b447fbddec2aaaaa077c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } } }
I20260928 02:42:55.701654 30642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2 [sys.catalog]: This master's current role is: LEADER
W20260928 02:42:55.702970 30656 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:42:55.703064 30656 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:42:55.703184 30647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:55.703447 30647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:42:55 dist-test-slave-kkrh krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:42:55.705617 30647 catalog_manager.cc:1347] Generated new cluster ID: 45e9a3e0311b452697f92161531e7a2f
I20260928 02:42:55.705668 30647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:55.709318 30640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:55.697681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:43341 (local address 127.0.0.1:42899)
0928 02:42:55.697841 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:55.697849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:55.697875 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:42:55.698836 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:55.698842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:55.698895 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:55.699001 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:55.699908 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:55.700914 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:55.701986 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:55.702784 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:55.702927 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:55.705741 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:55.705745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:55.705745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:55.705757 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:55.707621 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:55.708098 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:55.708101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:55.708102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:55.708216 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:55.708479 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:55.708480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:55.708480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:55.708737 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:55.708933 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:55.709034 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 02:42:55.709162 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260928 02:42:55.710306 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:42:55.714365 30647 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:55.715014 30647 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:55.718544 30640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:55.711697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:33713 (local address 127.0.0.1:42899)
0928 02:42:55.711738 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:55.711740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:55.711888 (+ 148us) server_negotiation.cc:407] Connection header received
0928 02:42:55.711966 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:55.711967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:55.711978 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:55.712044 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:55.712598 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:55.713157 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:55.714367 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:55.714656 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:55.714765 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:55.715993 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:55.715994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:55.715994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:55.716000 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:55.717401 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:55.717759 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:55.717760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:55.717760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:55.717816 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:55.718042 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:55.718043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:55.718043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:55.718149 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:55.718255 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:55.718382 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:42:55.718444 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:42:55.728555 30647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 686aaf6c350b447fbddec2aaaaa077c2: Generated new TSK 0
I20260928 02:42:55.728749 30647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:55.736310 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 30568
W20260928 02:42:55.740103 30553 connection.cc:535] client connection to 127.0.0.1:42899 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42899): Connection reset by peer (error 104)
2026-09-28T02:42:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:55 dist-test-slave-kkrh krb5kdc[30667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:55 dist-test-slave-kkrh krb5kdc[30667](info): set up 2 sockets
Sep 28 02:42:55 dist-test-slave-kkrh krb5kdc[30667](info): commencing operation
krb5kdc: starting...
W20260928 02:42:56.767899 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:42:56 dist-test-slave-kkrh krb5kdc[30667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:42:56.882270 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:56.960460 30680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:56.960652 30680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:56.960675 30680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:56.961803 30680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:56.961846 30680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:56.961860 30680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:56.961874 30680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:56.961933 30680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:56.963353 30680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:56.963601 30680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:56.963846 30680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:56.968586 30686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:56.968611 30688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:56.968626 30685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:56.968904 30680 server_base.cc:1034] running on GCE node
I20260928 02:42:56.969159 30680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:56.969424 30680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:56.970568 30680 hybrid_clock.cc:648] HybridClock initialized: now 1790563376970555 us; error 28 us; skew 500 ppm
Sep 28 02:42:56 dist-test-slave-kkrh krb5kdc[30667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:56.972947 30680 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:42:56.973417 30680 webserver.cc:460] Webserver started at http://127.0.0.1:43301/ using document root <none> and password file <none>
I20260928 02:42:56.973639 30680 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:56.973704 30680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:56.973852 30680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:56.974851 30680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "061fb87bf8884e309256ac4f1287e0f9"
format_stamp: "Formatted at 2026-09-28 02:42:56 on dist-test-slave-kkrh"
server_key: "c438227c821e3be0ba18a441f316bf23"
server_key_iv: "46c4a02aec62149aaf2cdd9a4ea9619a"
server_key_version: "encryptionkey@0"
I20260928 02:42:56.975219 30680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "061fb87bf8884e309256ac4f1287e0f9"
format_stamp: "Formatted at 2026-09-28 02:42:56 on dist-test-slave-kkrh"
server_key: "c438227c821e3be0ba18a441f316bf23"
server_key_iv: "46c4a02aec62149aaf2cdd9a4ea9619a"
server_key_version: "encryptionkey@0"
I20260928 02:42:56.976598 30680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:42:56.977270 30695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:56.977427 30680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:42:56.977523 30680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "061fb87bf8884e309256ac4f1287e0f9"
format_stamp: "Formatted at 2026-09-28 02:42:56 on dist-test-slave-kkrh"
server_key: "c438227c821e3be0ba18a441f316bf23"
server_key_iv: "46c4a02aec62149aaf2cdd9a4ea9619a"
server_key_version: "encryptionkey@0"
I20260928 02:42:56.977603 30680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:56.987188 30680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:56.988945 30680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:56.989104 30680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:56.992991 30680 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39867
I20260928 02:42:56.993021 30747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39867 every 8 connection(s)
I20260928 02:42:56.993360 30680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:56.994279 30748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:56.996284 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 30680
I20260928 02:42:56.996402 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:56.996587 9736 external_mini_cluster.cc:1428] Setting key ee120856a83411ca90328e6bd93c9509
I20260928 02:42:56.996785 30748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9: Bootstrap starting.
I20260928 02:42:56.997534 30748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:56.997843 30748 log.cc:824] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:56.998483 30748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9: No bootstrap required, opened a new log
I20260928 02:42:57.000284 30748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061fb87bf8884e309256ac4f1287e0f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39867 } }
I20260928 02:42:57.000411 30748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:57.000439 30748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 061fb87bf8884e309256ac4f1287e0f9, State: Initialized, Role: FOLLOWER
I20260928 02:42:57.000586 30748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061fb87bf8884e309256ac4f1287e0f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39867 } }
I20260928 02:42:57.000659 30748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:57.000681 30748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:57.000790 30748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:57.001493 30748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061fb87bf8884e309256ac4f1287e0f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39867 } }
I20260928 02:42:57.001613 30748 leader_election.cc:302] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 061fb87bf8884e309256ac4f1287e0f9; no voters:
I20260928 02:42:57.001768 30748 leader_election.cc:288] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:57.001801 30753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:57.002036 30753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [term 1 LEADER]: Becoming Leader. State: Replica: 061fb87bf8884e309256ac4f1287e0f9, State: Running, Role: LEADER
I20260928 02:42:57.002032 30748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:57.002180 30753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061fb87bf8884e309256ac4f1287e0f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39867 } }
I20260928 02:42:57.002342 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:56.997094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:37137 (local address 127.0.0.1:39867)
0928 02:42:56.997289 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:56.997299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:56.997331 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:42:56.997916 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:56.997926 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:56.997992 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:56.998135 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:56.998365 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:56.999293 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:57.000168 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:57.000692 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:57.000792 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:57.001536 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:57.001540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:57.001541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:57.001739 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:57.001778 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:57.001793 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:57.001924 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:57.002075 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:42:57.002104 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260928 02:42:57.002468 30753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 061fb87bf8884e309256ac4f1287e0f9. Latest consensus state: current_term: 1 leader_uuid: "061fb87bf8884e309256ac4f1287e0f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061fb87bf8884e309256ac4f1287e0f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39867 } } }
I20260928 02:42:57.002529 30753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:57.003096 30755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "061fb87bf8884e309256ac4f1287e0f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061fb87bf8884e309256ac4f1287e0f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39867 } } }
I20260928 02:42:57.003201 30755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:57.003283 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:42:57.003348 30759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:57.003969 30759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:57.005970 30759 catalog_manager.cc:1347] Generated new cluster ID: bf3f100418ad4fecb43fb702f434bd20
I20260928 02:42:57.006038 30759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:57.009277 30759 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:57.009940 30759 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:57.010218 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:57.004880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:35647 (local address 127.0.0.1:39867)
0928 02:42:57.004915 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:57.004918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:57.005055 (+ 137us) server_negotiation.cc:407] Connection header received
0928 02:42:57.005133 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:57.005134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:57.005143 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:57.005192 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:57.005339 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:57.005876 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:57.008838 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:57.009171 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:57.009250 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:57.009942 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:57.009944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:57.009944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:57.009948 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:57.009963 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:57.009968 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:57.010030 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:57.010142 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 02:42:57.010152 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:42:57.017158 30759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 061fb87bf8884e309256ac4f1287e0f9: Generated new TSK 0
I20260928 02:42:57.017428 30759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:57.023983 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 30680
W20260928 02:42:57.029356 30665 connection.cc:535] client connection to 127.0.0.1:39867 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39867): Connection reset by peer (error 104)
2026-09-28T02:42:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:57 dist-test-slave-kkrh krb5kdc[30779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:57 dist-test-slave-kkrh krb5kdc[30779](info): set up 2 sockets
Sep 28 02:42:57 dist-test-slave-kkrh krb5kdc[30779](info): commencing operation
krb5kdc: starting...
W20260928 02:42:58.056249 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:42:58 dist-test-slave-kkrh krb5kdc[30779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:42:58.170646 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:58.249853 30792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:58.250079 30792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:58.250115 30792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:58.251263 30792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:58.251327 30792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:58.251353 30792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:58.251379 30792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:58.251410 30792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:58.252869 30792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:58.253146 30792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:58.253360 30792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:58.258180 30800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:58.258206 30798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:58.258193 30797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:58.258332 30792 server_base.cc:1034] running on GCE node
I20260928 02:42:58.258678 30792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:58.258956 30792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:58.260114 30792 hybrid_clock.cc:648] HybridClock initialized: now 1790563378260110 us; error 36 us; skew 500 ppm
Sep 28 02:42:58 dist-test-slave-kkrh krb5kdc[30779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:58.266933 30792 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:42:58.267433 30792 webserver.cc:460] Webserver started at http://127.0.0.1:44557/ using document root <none> and password file <none>
I20260928 02:42:58.267669 30792 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:58.267738 30792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:58.267910 30792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:58.268951 30792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "e8019f07077c44fbaf8b8631472b578f"
format_stamp: "Formatted at 2026-09-28 02:42:58 on dist-test-slave-kkrh"
server_key: "e6729741e01b2be0c7ae4693eab280d2"
server_key_iv: "9e2a537e5d7e2b3881542ee3a3ee0ec3"
server_key_version: "encryptionkey@0"
I20260928 02:42:58.269325 30792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "e8019f07077c44fbaf8b8631472b578f"
format_stamp: "Formatted at 2026-09-28 02:42:58 on dist-test-slave-kkrh"
server_key: "e6729741e01b2be0c7ae4693eab280d2"
server_key_iv: "9e2a537e5d7e2b3881542ee3a3ee0ec3"
server_key_version: "encryptionkey@0"
I20260928 02:42:58.270574 30792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:42:58.271291 30807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:58.271456 30792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:42:58.271557 30792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "e8019f07077c44fbaf8b8631472b578f"
format_stamp: "Formatted at 2026-09-28 02:42:58 on dist-test-slave-kkrh"
server_key: "e6729741e01b2be0c7ae4693eab280d2"
server_key_iv: "9e2a537e5d7e2b3881542ee3a3ee0ec3"
server_key_version: "encryptionkey@0"
I20260928 02:42:58.271629 30792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:58.274852 30792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:58.276945 30792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:58.277136 30792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:58.282158 30792 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39233
I20260928 02:42:58.282186 30859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39233 every 8 connection(s)
I20260928 02:42:58.282550 30792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:58.283478 30860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:58.284942 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 30792
I20260928 02:42:58.285053 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:58.285259 9736 external_mini_cluster.cc:1428] Setting key cc58bd6bca3101caed846cb9c098aaf8
I20260928 02:42:58.286098 30860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f: Bootstrap starting.
I20260928 02:42:58.286652 30860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:58.286945 30860 log.cc:824] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:58.287844 30860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f: No bootstrap required, opened a new log
I20260928 02:42:58.288075 30863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:58.285811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:47579 (local address 127.0.0.1:39233)
0928 02:42:58.286020 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:58.286028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:58.286056 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:42:58.286568 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:58.286575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:58.286611 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:58.286784 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:58.287504 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:58.287508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:58.287508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:58.287513 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:58.287556 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:58.287570 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:58.287665 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:58.287846 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 02:42:58.287888 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260928 02:42:58.289119 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:42:58.290444 30860 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8019f07077c44fbaf8b8631472b578f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39233 } }
I20260928 02:42:58.290625 30860 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:58.290701 30860 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8019f07077c44fbaf8b8631472b578f, State: Initialized, Role: FOLLOWER
I20260928 02:42:58.290839 30860 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8019f07077c44fbaf8b8631472b578f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39233 } }
I20260928 02:42:58.290946 30860 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:58.290975 30860 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:58.291039 30860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:58.291697 30860 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8019f07077c44fbaf8b8631472b578f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39233 } }
I20260928 02:42:58.291813 30860 leader_election.cc:302] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8019f07077c44fbaf8b8631472b578f; no voters:
I20260928 02:42:58.291981 30863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:58.290681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:44643 (local address 127.0.0.1:39233)
0928 02:42:58.290721 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:58.290723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:58.290834 (+ 111us) server_negotiation.cc:407] Connection header received
0928 02:42:58.290928 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:58.290929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:58.290978 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:58.291036 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:58.291668 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:58.291670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:58.291670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:58.291675 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:58.291687 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:58.291691 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:58.291739 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:58.291810 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:42:58.291915 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:42:58.292003 30860 leader_election.cc:288] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:58.292030 30870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:58.292131 30870 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [term 1 LEADER]: Becoming Leader. State: Replica: e8019f07077c44fbaf8b8631472b578f, State: Running, Role: LEADER
I20260928 02:42:58.292230 30860 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:58.292413 30870 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8019f07077c44fbaf8b8631472b578f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39233 } }
I20260928 02:42:58.292795 30871 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8019f07077c44fbaf8b8631472b578f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8019f07077c44fbaf8b8631472b578f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39233 } } }
I20260928 02:42:58.292893 30871 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:58.293033 30871 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8019f07077c44fbaf8b8631472b578f. Latest consensus state: current_term: 1 leader_uuid: "e8019f07077c44fbaf8b8631472b578f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8019f07077c44fbaf8b8631472b578f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39233 } } }
I20260928 02:42:58.293084 30871 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:58.293107 30877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:58.293618 30877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:58.295077 30877 catalog_manager.cc:1347] Generated new cluster ID: 4cb938bbf6e3418a83df316d98cbe074
I20260928 02:42:58.295130 30877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:58.312234 30877 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:58.312381 30877 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:58.321463 30877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8019f07077c44fbaf8b8631472b578f: Generated new TSK 0
I20260928 02:42:58.321659 30877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:58.328102 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 30792
2026-09-28T02:42:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:58 dist-test-slave-kkrh krb5kdc[30891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:58 dist-test-slave-kkrh krb5kdc[30891](info): set up 2 sockets
Sep 28 02:42:58 dist-test-slave-kkrh krb5kdc[30891](info): commencing operation
krb5kdc: starting...
W20260928 02:42:59.359871 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:42:59 dist-test-slave-kkrh krb5kdc[30891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:42:59.470356 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:59.548600 30904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:59.548833 30904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:59.548870 30904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:59.550019 30904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:59.550103 30904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:59.550130 30904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:59.550158 30904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:59.550184 30904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:59.551626 30904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41017
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:42:59.551920 30904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:59.552137 30904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:59.556764 30910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:59.556799 30909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:59.556923 30904 server_base.cc:1034] running on GCE node
W20260928 02:42:59.556799 30912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:59.557224 30904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:59.557466 30904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:59.558611 30904 hybrid_clock.cc:648] HybridClock initialized: now 1790563379558590 us; error 37 us; skew 500 ppm
Sep 28 02:42:59 dist-test-slave-kkrh krb5kdc[30891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:59.560985 30904 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:42:59.561390 30904 webserver.cc:460] Webserver started at http://127.0.0.1:39483/ using document root <none> and password file <none>
I20260928 02:42:59.561625 30904 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:59.561693 30904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:59.561841 30904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:59.562907 30904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "2e14d0e5955e43f5817c03d270a39a25"
format_stamp: "Formatted at 2026-09-28 02:42:59 on dist-test-slave-kkrh"
server_key: "a5401e5070dc66e06f6b0848b74e9430"
server_key_iv: "5fda3b8eca91b3fd47133408c60c6959"
server_key_version: "encryptionkey@0"
I20260928 02:42:59.563263 30904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "2e14d0e5955e43f5817c03d270a39a25"
format_stamp: "Formatted at 2026-09-28 02:42:59 on dist-test-slave-kkrh"
server_key: "a5401e5070dc66e06f6b0848b74e9430"
server_key_iv: "5fda3b8eca91b3fd47133408c60c6959"
server_key_version: "encryptionkey@0"
I20260928 02:42:59.564651 30904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:42:59.565366 30919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:59.565557 30904 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:42:59.565670 30904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "2e14d0e5955e43f5817c03d270a39a25"
format_stamp: "Formatted at 2026-09-28 02:42:59 on dist-test-slave-kkrh"
server_key: "a5401e5070dc66e06f6b0848b74e9430"
server_key_iv: "5fda3b8eca91b3fd47133408c60c6959"
server_key_version: "encryptionkey@0"
I20260928 02:42:59.565742 30904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:59.582188 30904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:59.584136 30904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:59.584296 30904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:59.588186 30904 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41017
I20260928 02:42:59.588222 30971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41017 every 8 connection(s)
I20260928 02:42:59.588563 30904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:59.589530 30972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:59.591698 30972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25: Bootstrap starting.
I20260928 02:42:59.592283 30972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:59.592571 30972 log.cc:824] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:59.593266 30972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25: No bootstrap required, opened a new log
I20260928 02:42:59.594687 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 30904
I20260928 02:42:59.594810 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:42:59.595012 9736 external_mini_cluster.cc:1428] Setting key 8f6a347a5af64cca454122629d64be1a
I20260928 02:42:59.595049 30972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e14d0e5955e43f5817c03d270a39a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41017 } }
I20260928 02:42:59.595286 30972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:59.595345 30972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e14d0e5955e43f5817c03d270a39a25, State: Initialized, Role: FOLLOWER
I20260928 02:42:59.595552 30972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e14d0e5955e43f5817c03d270a39a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41017 } }
I20260928 02:42:59.595690 30972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:59.595729 30972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:59.595853 30972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:59.596958 30972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e14d0e5955e43f5817c03d270a39a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41017 } }
I20260928 02:42:59.597072 30972 leader_election.cc:302] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e14d0e5955e43f5817c03d270a39a25; no voters:
I20260928 02:42:59.597232 30972 leader_election.cc:288] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:59.597306 30977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:59.597460 30972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:59.597468 30977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [term 1 LEADER]: Becoming Leader. State: Replica: 2e14d0e5955e43f5817c03d270a39a25, State: Running, Role: LEADER
I20260928 02:42:59.597709 30977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e14d0e5955e43f5817c03d270a39a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41017 } }
I20260928 02:42:59.598115 30978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e14d0e5955e43f5817c03d270a39a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e14d0e5955e43f5817c03d270a39a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41017 } } }
I20260928 02:42:59.598099 30979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e14d0e5955e43f5817c03d270a39a25. Latest consensus state: current_term: 1 leader_uuid: "2e14d0e5955e43f5817c03d270a39a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e14d0e5955e43f5817c03d270a39a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41017 } } }
I20260928 02:42:59.598217 30978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:59.598222 30979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:59.598510 30986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:59.599222 30986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:59.601118 30986 catalog_manager.cc:1347] Generated new cluster ID: 992d96afecab4041b8c3b488f4127750
I20260928 02:42:59.601172 30986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:42:59 dist-test-slave-kkrh krb5kdc[30891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:42:59.607345 30976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:59.595617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:46911 (local address 127.0.0.1:41017)
0928 02:42:59.595867 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:59.595877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:59.595907 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:42:59.596059 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:59.596065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:59.596109 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:59.596243 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:59.597125 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:59.598534 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:59.599644 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:59.600205 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:59.600312 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:59.603257 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:59.603260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:59.603261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:59.603271 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:59.605683 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:59.606140 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:59.606143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:59.606143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:59.606255 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:59.606506 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:59.606507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:59.606507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:59.606777 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:59.606914 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:59.607063 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 02:42:59.607154 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":125,"threads_started":1}
I20260928 02:42:59.608295 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:42:59.616839 30976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:59.609665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:45351 (local address 127.0.0.1:41017)
0928 02:42:59.609723 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:59.609724 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:59.609913 (+ 189us) server_negotiation.cc:407] Connection header received
0928 02:42:59.609969 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:59.609970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:59.609980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:59.610043 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:59.610624 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:59.611344 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:59.612409 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:59.612718 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:59.612776 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:59.614162 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:59.614163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:59.614163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:59.614169 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:59.615630 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:59.616065 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:59.616066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:59.616067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:59.616121 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:59.616361 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:59.616361 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:59.616362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:59.616470 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:59.616570 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:59.616705 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 02:42:59.616759 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:42:59.621045 30986 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:59.621538 30986 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:59.625339 30986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e14d0e5955e43f5817c03d270a39a25: Generated new TSK 0
I20260928 02:42:59.625519 30986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:59.632592 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 30904
W20260928 02:42:59.636323 30889 connection.cc:535] client connection to 127.0.0.1:41017 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41017): Connection reset by peer (error 104)
2026-09-28T02:42:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:59 dist-test-slave-kkrh krb5kdc[31003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:59 dist-test-slave-kkrh krb5kdc[31003](info): set up 2 sockets
Sep 28 02:42:59 dist-test-slave-kkrh krb5kdc[31003](info): commencing operation
krb5kdc: starting...
W20260928 02:43:00.699772 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:43:00 dist-test-slave-kkrh krb5kdc[31003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:43:00.814646 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:00.894413 31016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:00.894604 31016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:00.894626 31016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:00.895675 31016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:00.895707 31016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:00.895722 31016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:00.895735 31016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:00.895746 31016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:00.897113 31016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:00.897287 31016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:00.897439 31016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:00.902292 31022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:00.902282 31024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:00.902329 31021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:00.902498 31016 server_base.cc:1034] running on GCE node
I20260928 02:43:00.902714 31016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:00.902951 31016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:00.904091 31016 hybrid_clock.cc:648] HybridClock initialized: now 1790563380904069 us; error 34 us; skew 500 ppm
Sep 28 02:43:00 dist-test-slave-kkrh krb5kdc[31003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:00.906503 31016 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:43:00.906929 31016 webserver.cc:460] Webserver started at http://127.0.0.1:37531/ using document root <none> and password file <none>
I20260928 02:43:00.907138 31016 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:00.907202 31016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:00.907346 31016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:00.908437 31016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "d23ee7d5865646b58e99a385b8e426ae"
format_stamp: "Formatted at 2026-09-28 02:43:00 on dist-test-slave-kkrh"
server_key: "9ae5e99dcdd9c65df1476e201488f55c"
server_key_iv: "5701843e2f24ed1089aac4e258687b54"
server_key_version: "encryptionkey@0"
I20260928 02:43:00.908788 31016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "d23ee7d5865646b58e99a385b8e426ae"
format_stamp: "Formatted at 2026-09-28 02:43:00 on dist-test-slave-kkrh"
server_key: "9ae5e99dcdd9c65df1476e201488f55c"
server_key_iv: "5701843e2f24ed1089aac4e258687b54"
server_key_version: "encryptionkey@0"
I20260928 02:43:00.910056 31016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:00.910753 31031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:00.910921 31016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:00.911013 31016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "d23ee7d5865646b58e99a385b8e426ae"
format_stamp: "Formatted at 2026-09-28 02:43:00 on dist-test-slave-kkrh"
server_key: "9ae5e99dcdd9c65df1476e201488f55c"
server_key_iv: "5701843e2f24ed1089aac4e258687b54"
server_key_version: "encryptionkey@0"
I20260928 02:43:00.911089 31016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:00.920960 31016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:00.922662 31016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:00.922812 31016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:00.926652 31016 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43753
I20260928 02:43:00.926685 31083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43753 every 8 connection(s)
I20260928 02:43:00.927007 31016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:00.927778 31084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:00.929085 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 31016
I20260928 02:43:00.929207 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:43:00.929391 9736 external_mini_cluster.cc:1428] Setting key b0cfc3b7e7f3ec77db6d440a3ea2df76
I20260928 02:43:00.930451 31084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae: Bootstrap starting.
I20260928 02:43:00.931130 31084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:00.931411 31084 log.cc:824] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:00.932147 31084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae: No bootstrap required, opened a new log
I20260928 02:43:00.934474 31084 raft_consensus.cc:348] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23ee7d5865646b58e99a385b8e426ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43753 } }
I20260928 02:43:00.934599 31084 raft_consensus.cc:374] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:00.934620 31084 raft_consensus.cc:729] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23ee7d5865646b58e99a385b8e426ae, State: Initialized, Role: FOLLOWER
I20260928 02:43:00.934746 31084 consensus_queue.cc:260] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23ee7d5865646b58e99a385b8e426ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43753 } }
I20260928 02:43:00.934824 31084 raft_consensus.cc:388] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:00.934882 31084 raft_consensus.cc:482] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:00.934953 31084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:00.935129 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:00.929900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:40107 (local address 127.0.0.1:43753)
0928 02:43:00.930069 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:00.930077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:00.930110 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:43:00.930526 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:00.930534 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:00.930602 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:00.930712 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:00.930982 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:00.932128 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:00.933232 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:00.933624 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:00.933705 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:00.934426 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:00.934429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:00.934429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:00.934583 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:00.934642 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:00.934658 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:00.934765 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:00.934888 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 02:43:00.934916 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260928 02:43:00.935624 31084 raft_consensus.cc:504] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23ee7d5865646b58e99a385b8e426ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43753 } }
I20260928 02:43:00.935732 31084 leader_election.cc:302] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d23ee7d5865646b58e99a385b8e426ae; no voters:
I20260928 02:43:00.935914 31084 leader_election.cc:288] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:00.935940 31089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:00.936062 31089 raft_consensus.cc:686] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [term 1 LEADER]: Becoming Leader. State: Replica: d23ee7d5865646b58e99a385b8e426ae, State: Running, Role: LEADER
I20260928 02:43:00.936136 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:00.936161 31084 sys_catalog.cc:564] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:00.936193 31089 consensus_queue.cc:237] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23ee7d5865646b58e99a385b8e426ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43753 } }
I20260928 02:43:00.936501 31089 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader d23ee7d5865646b58e99a385b8e426ae. Latest consensus state: current_term: 1 leader_uuid: "d23ee7d5865646b58e99a385b8e426ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23ee7d5865646b58e99a385b8e426ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43753 } } }
I20260928 02:43:00.936514 31090 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d23ee7d5865646b58e99a385b8e426ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23ee7d5865646b58e99a385b8e426ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43753 } } }
I20260928 02:43:00.936650 31090 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:00.936846 31089 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:00.937014 31096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:00.937516 31096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:00.939021 31096 catalog_manager.cc:1347] Generated new cluster ID: 057a4cd5e19a4ebdaa5a8df1d63fc3fe
I20260928 02:43:00.939077 31096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:00.943094 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:00.940206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:55941 (local address 127.0.0.1:43753)
0928 02:43:00.940240 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:00.940242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:00.940420 (+ 178us) server_negotiation.cc:407] Connection header received
0928 02:43:00.940492 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:00.940493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:00.940505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:00.940560 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:00.940680 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:00.941167 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:00.941886 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:00.942107 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:00.942183 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:00.942828 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:00.942829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:00.942830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:00.942834 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:00.942847 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:00.942851 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:00.942907 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:00.942992 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:43:00.943001 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:43:00.959064 31096 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:00.959625 31096 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:00.966444 31096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d23ee7d5865646b58e99a385b8e426ae: Generated new TSK 0
I20260928 02:43:00.966643 31096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:00.976279 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 31016
W20260928 02:43:00.980167 31001 connection.cc:535] client connection to 127.0.0.1:43753 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43753): Connection reset by peer (error 104)
2026-09-28T02:43:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:01 dist-test-slave-kkrh krb5kdc[31115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:01 dist-test-slave-kkrh krb5kdc[31115](info): set up 2 sockets
Sep 28 02:43:01 dist-test-slave-kkrh krb5kdc[31115](info): commencing operation
krb5kdc: starting...
W20260928 02:43:02.004033 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:43:02 dist-test-slave-kkrh krb5kdc[31115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:43:02.118837 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:02.197342 31128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:02.197528 31128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:02.197551 31128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:02.198683 31128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:02.198724 31128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:02.198738 31128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:02.198776 31128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:02.198807 31128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:02.200301 31128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44245
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:02.200552 31128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:02.200765 31128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:02.205390 31134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:02.205466 31133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:02.205389 31136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:02.205467 31128 server_base.cc:1034] running on GCE node
I20260928 02:43:02.205827 31128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:02.206063 31128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:02.207226 31128 hybrid_clock.cc:648] HybridClock initialized: now 1790563382207216 us; error 39 us; skew 500 ppm
Sep 28 02:43:02 dist-test-slave-kkrh krb5kdc[31115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:02.209771 31128 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:43:02.210233 31128 webserver.cc:460] Webserver started at http://127.0.0.1:46875/ using document root <none> and password file <none>
I20260928 02:43:02.210494 31128 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:02.210556 31128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:02.210690 31128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:02.211683 31128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "d71d46950b594f78b776daef2250de6f"
format_stamp: "Formatted at 2026-09-28 02:43:02 on dist-test-slave-kkrh"
server_key: "72dab1a60a3a2a020efe9f749ee8d49d"
server_key_iv: "2504294a2ab9a0dded6dddca6b61d4a1"
server_key_version: "encryptionkey@0"
I20260928 02:43:02.212066 31128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "d71d46950b594f78b776daef2250de6f"
format_stamp: "Formatted at 2026-09-28 02:43:02 on dist-test-slave-kkrh"
server_key: "72dab1a60a3a2a020efe9f749ee8d49d"
server_key_iv: "2504294a2ab9a0dded6dddca6b61d4a1"
server_key_version: "encryptionkey@0"
I20260928 02:43:02.213333 31128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:02.214022 31143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:02.214200 31128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:02.214295 31128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "d71d46950b594f78b776daef2250de6f"
format_stamp: "Formatted at 2026-09-28 02:43:02 on dist-test-slave-kkrh"
server_key: "72dab1a60a3a2a020efe9f749ee8d49d"
server_key_iv: "2504294a2ab9a0dded6dddca6b61d4a1"
server_key_version: "encryptionkey@0"
I20260928 02:43:02.214391 31128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:02.217633 31128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:02.219923 31128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:02.220067 31128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:02.223966 31128 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44245
I20260928 02:43:02.223994 31195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44245 every 8 connection(s)
I20260928 02:43:02.224303 31128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:02.225210 31196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:02.227234 31196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f: Bootstrap starting.
I20260928 02:43:02.227770 31196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:02.228060 31196 log.cc:824] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:02.228645 31196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f: No bootstrap required, opened a new log
I20260928 02:43:02.230163 31196 raft_consensus.cc:348] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71d46950b594f78b776daef2250de6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44245 } }
I20260928 02:43:02.230289 31196 raft_consensus.cc:374] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:02.230309 31196 raft_consensus.cc:729] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71d46950b594f78b776daef2250de6f, State: Initialized, Role: FOLLOWER
I20260928 02:43:02.230400 31196 consensus_queue.cc:260] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71d46950b594f78b776daef2250de6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44245 } }
I20260928 02:43:02.230450 31196 raft_consensus.cc:388] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:02.230469 31196 raft_consensus.cc:482] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:02.230536 31196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:02.231027 31196 raft_consensus.cc:504] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71d46950b594f78b776daef2250de6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44245 } }
I20260928 02:43:02.231133 31196 leader_election.cc:302] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d71d46950b594f78b776daef2250de6f; no voters:
I20260928 02:43:02.231302 31196 leader_election.cc:288] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:02.231356 31199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:02.231556 31199 raft_consensus.cc:686] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [term 1 LEADER]: Becoming Leader. State: Replica: d71d46950b594f78b776daef2250de6f, State: Running, Role: LEADER
I20260928 02:43:02.231596 31196 sys_catalog.cc:564] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:02.231717 31199 consensus_queue.cc:237] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71d46950b594f78b776daef2250de6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44245 } }
I20260928 02:43:02.232147 31200 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d71d46950b594f78b776daef2250de6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71d46950b594f78b776daef2250de6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44245 } } }
I20260928 02:43:02.232164 31201 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d71d46950b594f78b776daef2250de6f. Latest consensus state: current_term: 1 leader_uuid: "d71d46950b594f78b776daef2250de6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71d46950b594f78b776daef2250de6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44245 } } }
I20260928 02:43:02.232311 31200 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:02.232333 31201 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:02.232642 31204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:02.233096 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 31128
I20260928 02:43:02.233171 31204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:02.233237 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:43:02.233428 9736 external_mini_cluster.cc:1428] Setting key 58f09b8c2010002824d4b55eb4c2feb7
I20260928 02:43:02.234817 31204 catalog_manager.cc:1347] Generated new cluster ID: 1be3eb2e0d6c4235b1371076b9a087fc
I20260928 02:43:02.234866 31204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:02.235591 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:02.233843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:43023 (local address 127.0.0.1:44245)
0928 02:43:02.234010 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:02.234019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:02.234049 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:43:02.234215 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:02.234221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:02.234251 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:02.234398 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:02.235096 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:02.235100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:02.235100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:02.235104 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:02.235147 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:02.235161 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:02.235213 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:02.235382 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:43:02.235424 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260928 02:43:02.240852 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:02.240875 31204 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:02.241003 31204 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:02.243381 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:02.242366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:50623 (local address 127.0.0.1:44245)
0928 02:43:02.242403 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:02.242405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:02.242500 (+ 95us) server_negotiation.cc:407] Connection header received
0928 02:43:02.242563 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:02.242564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:02.242568 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:02.242615 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:02.243173 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:02.243173 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:02.243174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:02.243178 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:02.243191 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:02.243196 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:02.243250 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:02.243309 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:43:02.243322 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:43:02.246028 31204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d71d46950b594f78b776daef2250de6f: Generated new TSK 0
I20260928 02:43:02.246181 31204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:02.250653 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 31128
2026-09-28T02:43:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1274 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:02 dist-test-slave-kkrh krb5kdc[31227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:02 dist-test-slave-kkrh krb5kdc[31227](info): set up 2 sockets
Sep 28 02:43:02 dist-test-slave-kkrh krb5kdc[31227](info): commencing operation
krb5kdc: starting...
W20260928 02:43:03.307888 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:43:03 dist-test-slave-kkrh krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:43:03.423977 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:33385
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:33385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:03.502024 31240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:03.502246 31240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:03.502285 31240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:03.503407 31240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:03.503474 31240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:03.503499 31240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:03.503525 31240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:03.503557 31240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:03.505030 31240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:33385
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:33385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:03.505295 31240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:03.505512 31240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:43:03.510352 31240 server_base.cc:1034] running on GCE node
W20260928 02:43:03.510285 31246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:03.510283 31248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:03.510344 31245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:03.510784 31240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:03.511060 31240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:03.512224 31240 hybrid_clock.cc:648] HybridClock initialized: now 1790563383512215 us; error 39 us; skew 500 ppm
Sep 28 02:43:03 dist-test-slave-kkrh krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563383, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:03.518769 31240 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:43:03.519215 31240 webserver.cc:460] Webserver started at http://127.9.130.62:43193/ using document root <none> and password file <none>
I20260928 02:43:03.519449 31240 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:03.519519 31240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:03.519665 31240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:03.520776 31240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "6fb253eeb28244bd8e56ee20ea363a6c"
format_stamp: "Formatted at 2026-09-28 02:43:03 on dist-test-slave-kkrh"
server_key: "07b879c50fd24322081d4241d52406dd"
server_key_iv: "477fab60f61f3b230ef021735b58c101"
server_key_version: "encryptionkey@0"
I20260928 02:43:03.521168 31240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "6fb253eeb28244bd8e56ee20ea363a6c"
format_stamp: "Formatted at 2026-09-28 02:43:03 on dist-test-slave-kkrh"
server_key: "07b879c50fd24322081d4241d52406dd"
server_key_iv: "477fab60f61f3b230ef021735b58c101"
server_key_version: "encryptionkey@0"
I20260928 02:43:03.522804 31240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:03.523524 31255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:03.523711 31240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:03.523805 31240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "6fb253eeb28244bd8e56ee20ea363a6c"
format_stamp: "Formatted at 2026-09-28 02:43:03 on dist-test-slave-kkrh"
server_key: "07b879c50fd24322081d4241d52406dd"
server_key_iv: "477fab60f61f3b230ef021735b58c101"
server_key_version: "encryptionkey@0"
I20260928 02:43:03.523914 31240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:03.532320 31240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:03.534523 31240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:03.534675 31240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:03.538805 31240 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:33385
I20260928 02:43:03.538834 31307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:33385 every 8 connection(s)
I20260928 02:43:03.539175 31240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:03.540117 31308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:03.542299 31308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c: Bootstrap starting.
I20260928 02:43:03.542881 31308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:03.543154 31308 log.cc:824] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:03.543949 31308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c: No bootstrap required, opened a new log
I20260928 02:43:03.545495 31308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb253eeb28244bd8e56ee20ea363a6c" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33385 } }
I20260928 02:43:03.545629 31308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:03.545652 31308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb253eeb28244bd8e56ee20ea363a6c, State: Initialized, Role: FOLLOWER
I20260928 02:43:03.545812 31308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb253eeb28244bd8e56ee20ea363a6c" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33385 } }
I20260928 02:43:03.545897 31308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:03.545948 31308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:03.545998 31308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:03.546788 31308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb253eeb28244bd8e56ee20ea363a6c" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33385 } }
I20260928 02:43:03.546896 31308 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fb253eeb28244bd8e56ee20ea363a6c; no voters:
I20260928 02:43:03.547075 31308 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:03.547117 31311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:03.547291 31311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [term 1 LEADER]: Becoming Leader. State: Replica: 6fb253eeb28244bd8e56ee20ea363a6c, State: Running, Role: LEADER
I20260928 02:43:03.547333 31308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:03.547410 31311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb253eeb28244bd8e56ee20ea363a6c" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33385 } }
I20260928 02:43:03.547834 31312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fb253eeb28244bd8e56ee20ea363a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb253eeb28244bd8e56ee20ea363a6c" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33385 } } }
I20260928 02:43:03.548002 31312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:03.548105 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 31240
I20260928 02:43:03.548230 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:43:03.548447 9736 external_mini_cluster.cc:1428] Setting key 2d9253ef25f869082237686bff0e2cf7
I20260928 02:43:03.548456 31320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:03.547845 31313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb253eeb28244bd8e56ee20ea363a6c. Latest consensus state: current_term: 1 leader_uuid: "6fb253eeb28244bd8e56ee20ea363a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb253eeb28244bd8e56ee20ea363a6c" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33385 } } }
I20260928 02:43:03.548615 31313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:03.548997 31320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:03.550577 31320 catalog_manager.cc:1347] Generated new cluster ID: f9245f773c5b4df08b3e60397672d596
I20260928 02:43:03.550632 31320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:43:03 dist-test-slave-kkrh krb5kdc[31227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:43:03.558436 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:03.548961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33317 (local address 127.9.130.62:33385)
0928 02:43:03.549144 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:03.549154 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:03.549436 (+ 282us) server_negotiation.cc:407] Connection header received
0928 02:43:03.549602 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:03.549609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:03.549662 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:03.549791 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:03.550543 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:03.551497 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:03.552356 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:03.552778 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:03.552903 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:03.555170 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:03.555173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:03.555174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:03.555185 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:03.556887 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:03.557284 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:03.557287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:03.557288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:03.557391 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:03.557621 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:03.557622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:03.557623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:03.557852 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:03.558028 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:03.558110 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 02:43:03.558215 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260928 02:43:03.559464 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:03.567255 31320 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:03.567998 31320 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:03.572785 31320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fb253eeb28244bd8e56ee20ea363a6c: Generated new TSK 0
I20260928 02:43:03.572986 31320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:03.573421 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:03.564143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57779 (local address 127.9.130.62:33385)
0928 02:43:03.564182 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:03.564184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:03.567278 (+ 3094us) server_negotiation.cc:407] Connection header received
0928 02:43:03.567301 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:03.567302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:03.567311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:03.567359 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:03.567971 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:03.568467 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:03.569197 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:03.569404 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:03.569486 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:03.570682 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:03.570683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:03.570684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:03.570690 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:03.572170 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:03.572568 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:03.572569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:03.572569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:03.572632 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:03.572854 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:03.572855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:03.572855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:03.572975 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:03.573088 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:03.573268 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 02:43:03.573335 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:43:03.575050 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 31240
W20260928 02:43:03.579022 31225 connection.cc:535] client connection to 127.9.130.62:33385 recv error: Network error: failed to read from TLS socket (remote: 127.9.130.62:33385): Connection reset by peer (error 104)
2026-09-28T02:43:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:03 dist-test-slave-kkrh krb5kdc[31339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:03 dist-test-slave-kkrh krb5kdc[31339](info): set up 2 sockets
Sep 28 02:43:03 dist-test-slave-kkrh krb5kdc[31339](info): commencing operation
krb5kdc: starting...
W20260928 02:43:04.623888 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:43:04 dist-test-slave-kkrh krb5kdc[31339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:43:04.736779 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:34847
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:34847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:04.815488 31352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:04.815675 31352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:04.815730 31352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:04.816927 31352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:04.816977 31352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:04.816992 31352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:04.817005 31352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:04.817016 31352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:04.818368 31352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:34847
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:04.818549 31352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:04.818706 31352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:04.823304 31358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:04.823287 31357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:04.823305 31360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:04.823590 31352 server_base.cc:1034] running on GCE node
I20260928 02:43:04.823904 31352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:04.824191 31352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:04.825352 31352 hybrid_clock.cc:648] HybridClock initialized: now 1790563384825342 us; error 42 us; skew 500 ppm
Sep 28 02:43:04 dist-test-slave-kkrh krb5kdc[31339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563384, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:04.827775 31352 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:43:04.828219 31352 webserver.cc:460] Webserver started at http://127.9.130.62:45269/ using document root <none> and password file <none>
I20260928 02:43:04.828430 31352 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:04.828480 31352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:04.828640 31352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:04.829638 31352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "0ec7c35c13414e3784cbc974e81acf96"
format_stamp: "Formatted at 2026-09-28 02:43:04 on dist-test-slave-kkrh"
server_key: "ee90a8396cf214729e8c9e01e72bdc5e"
server_key_iv: "7faeaca0310dffc1494e0dcb5a839b00"
server_key_version: "encryptionkey@0"
I20260928 02:43:04.829982 31352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "0ec7c35c13414e3784cbc974e81acf96"
format_stamp: "Formatted at 2026-09-28 02:43:04 on dist-test-slave-kkrh"
server_key: "ee90a8396cf214729e8c9e01e72bdc5e"
server_key_iv: "7faeaca0310dffc1494e0dcb5a839b00"
server_key_version: "encryptionkey@0"
I20260928 02:43:04.831265 31352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:43:04.832015 31367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:04.832201 31352 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:04.832286 31352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "0ec7c35c13414e3784cbc974e81acf96"
format_stamp: "Formatted at 2026-09-28 02:43:04 on dist-test-slave-kkrh"
server_key: "ee90a8396cf214729e8c9e01e72bdc5e"
server_key_iv: "7faeaca0310dffc1494e0dcb5a839b00"
server_key_version: "encryptionkey@0"
I20260928 02:43:04.832357 31352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:04.844885 31352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:04.847175 31352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:04.847337 31352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:04.851276 31352 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:34847
I20260928 02:43:04.851328 31419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:34847 every 8 connection(s)
I20260928 02:43:04.851642 31352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:04.852550 31420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:04.854668 31420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96: Bootstrap starting.
I20260928 02:43:04.855189 31420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:04.855413 31420 log.cc:824] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:04.856112 31420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96: No bootstrap required, opened a new log
I20260928 02:43:04.857569 31420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec7c35c13414e3784cbc974e81acf96" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34847 } }
I20260928 02:43:04.857700 31420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:04.857722 31420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec7c35c13414e3784cbc974e81acf96, State: Initialized, Role: FOLLOWER
I20260928 02:43:04.857851 31420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec7c35c13414e3784cbc974e81acf96" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34847 } }
I20260928 02:43:04.857936 31420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:04.857965 31420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:04.857993 31420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:04.858726 31420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec7c35c13414e3784cbc974e81acf96" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34847 } }
I20260928 02:43:04.858806 31420 leader_election.cc:302] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ec7c35c13414e3784cbc974e81acf96; no voters:
I20260928 02:43:04.858913 31420 leader_election.cc:288] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:04.858992 31423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:04.859179 31423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [term 1 LEADER]: Becoming Leader. State: Replica: 0ec7c35c13414e3784cbc974e81acf96, State: Running, Role: LEADER
I20260928 02:43:04.859194 31420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:04.859300 31423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec7c35c13414e3784cbc974e81acf96" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34847 } }
I20260928 02:43:04.859671 31424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ec7c35c13414e3784cbc974e81acf96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec7c35c13414e3784cbc974e81acf96" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34847 } } }
I20260928 02:43:04.859675 31425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ec7c35c13414e3784cbc974e81acf96. Latest consensus state: current_term: 1 leader_uuid: "0ec7c35c13414e3784cbc974e81acf96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec7c35c13414e3784cbc974e81acf96" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 34847 } } }
I20260928 02:43:04.859838 31424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:04.859853 31425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:04.860154 31432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:04.860684 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 31352
I20260928 02:43:04.860711 31432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:04.860807 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:43:04.861023 9736 external_mini_cluster.cc:1428] Setting key c4ba821346d83e58b4a6b42bcd01f674
I20260928 02:43:04.862428 31432 catalog_manager.cc:1347] Generated new cluster ID: 19ab526fefdd425aafd17cfb5a1d3253
I20260928 02:43:04.862475 31432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:04.866453 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.861495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52163 (local address 127.9.130.62:34847)
0928 02:43:04.861696 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:04.861706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:04.861796 (+ 90us) server_negotiation.cc:407] Connection header received
0928 02:43:04.861983 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:04.861989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:04.862042 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:04.862180 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:04.862434 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.863416 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.864397 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.864835 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.864973 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.865753 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:04.865757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:04.865758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:04.865953 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:04.865993 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:04.866005 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:04.866108 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:04.866196 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 02:43:04.866226 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260928 02:43:04.867835 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:04.872337 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.869295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41073 (local address 127.9.130.62:34847)
0928 02:43:04.869335 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:04.869337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:04.869409 (+ 72us) server_negotiation.cc:407] Connection header received
0928 02:43:04.869487 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:04.869488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:04.869499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:04.869560 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:04.869697 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.870193 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.870939 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.871191 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.871292 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.872045 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:04.872046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:04.872046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:04.872051 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:04.872066 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:04.872070 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:04.872135 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:04.872218 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:43:04.872231 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":19}
I20260928 02:43:04.877076 31432 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:04.877549 31432 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:04.880702 31432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ec7c35c13414e3784cbc974e81acf96: Generated new TSK 0
I20260928 02:43:04.880879 31432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:04.887750 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 31352
W20260928 02:43:04.891793 31337 connection.cc:535] client connection to 127.9.130.62:34847 recv error: Network error: failed to read from TLS socket (remote: 127.9.130.62:34847): Connection reset by peer (error 104)
2026-09-28T02:43:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:04 dist-test-slave-kkrh krb5kdc[31451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:04 dist-test-slave-kkrh krb5kdc[31451](info): set up 2 sockets
Sep 28 02:43:04 dist-test-slave-kkrh krb5kdc[31451](info): commencing operation
krb5kdc: starting...
W20260928 02:43:05.928120 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:43:06 dist-test-slave-kkrh krb5kdc[31451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:43:06.046159 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:36845
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:36845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:06.127013 31464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:06.127202 31464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:06.127223 31464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:06.128439 31464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:06.128480 31464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:06.128495 31464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:06.128528 31464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:06.128556 31464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:06.129988 31464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:36845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:36845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:06.130246 31464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:06.130482 31464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:06.135156 31469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:06.135179 31472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:06.135185 31470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:06.135416 31464 server_base.cc:1034] running on GCE node
I20260928 02:43:06.135681 31464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:06.135984 31464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:06.137126 31464 hybrid_clock.cc:648] HybridClock initialized: now 1790563386137112 us; error 29 us; skew 500 ppm
Sep 28 02:43:06 dist-test-slave-kkrh krb5kdc[31451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563386, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:06.139742 31464 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:43:06.140229 31464 webserver.cc:460] Webserver started at http://127.9.130.62:33865/ using document root <none> and password file <none>
I20260928 02:43:06.140455 31464 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:06.140522 31464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:06.140651 31464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:06.141631 31464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "577e70b39d2a437b9b7fa3bb9f185733"
format_stamp: "Formatted at 2026-09-28 02:43:06 on dist-test-slave-kkrh"
server_key: "1a543483e22a6b23c9c3366b6f1358d7"
server_key_iv: "579ec3e83b07fcfb4a4b65007d39a252"
server_key_version: "encryptionkey@0"
I20260928 02:43:06.141993 31464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "577e70b39d2a437b9b7fa3bb9f185733"
format_stamp: "Formatted at 2026-09-28 02:43:06 on dist-test-slave-kkrh"
server_key: "1a543483e22a6b23c9c3366b6f1358d7"
server_key_iv: "579ec3e83b07fcfb4a4b65007d39a252"
server_key_version: "encryptionkey@0"
I20260928 02:43:06.143307 31464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:06.144141 31479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:06.144335 31464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:06.144436 31464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "577e70b39d2a437b9b7fa3bb9f185733"
format_stamp: "Formatted at 2026-09-28 02:43:06 on dist-test-slave-kkrh"
server_key: "1a543483e22a6b23c9c3366b6f1358d7"
server_key_iv: "579ec3e83b07fcfb4a4b65007d39a252"
server_key_version: "encryptionkey@0"
I20260928 02:43:06.144520 31464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:06.147626 31464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:06.149365 31464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:06.149513 31464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:06.153439 31464 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:36845
I20260928 02:43:06.153469 31531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:36845 every 8 connection(s)
I20260928 02:43:06.153774 31464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:06.154708 31532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:06.156914 31532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733: Bootstrap starting.
I20260928 02:43:06.157465 31532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:06.157738 31532 log.cc:824] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:06.158403 31532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733: No bootstrap required, opened a new log
I20260928 02:43:06.160007 31532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577e70b39d2a437b9b7fa3bb9f185733" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36845 } }
I20260928 02:43:06.160133 31532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:06.160192 31532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577e70b39d2a437b9b7fa3bb9f185733, State: Initialized, Role: FOLLOWER
I20260928 02:43:06.160375 31532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577e70b39d2a437b9b7fa3bb9f185733" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36845 } }
I20260928 02:43:06.160478 31532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:06.160514 31532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:06.160512 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 31464
I20260928 02:43:06.160573 31532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:06.160609 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:43:06.160794 9736 external_mini_cluster.cc:1428] Setting key 307e1ea9c8004109e3e91c41453972fd
I20260928 02:43:06.161715 31532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577e70b39d2a437b9b7fa3bb9f185733" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36845 } }
I20260928 02:43:06.161823 31532 leader_election.cc:302] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 577e70b39d2a437b9b7fa3bb9f185733; no voters:
I20260928 02:43:06.162027 31532 leader_election.cc:288] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:06.162075 31537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:06.162520 31537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [term 1 LEADER]: Becoming Leader. State: Replica: 577e70b39d2a437b9b7fa3bb9f185733, State: Running, Role: LEADER
I20260928 02:43:06.162629 31532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:06.162680 31537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577e70b39d2a437b9b7fa3bb9f185733" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36845 } }
I20260928 02:43:06.163071 31539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "577e70b39d2a437b9b7fa3bb9f185733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577e70b39d2a437b9b7fa3bb9f185733" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36845 } } }
I20260928 02:43:06.163061 31537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 577e70b39d2a437b9b7fa3bb9f185733. Latest consensus state: current_term: 1 leader_uuid: "577e70b39d2a437b9b7fa3bb9f185733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577e70b39d2a437b9b7fa3bb9f185733" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 36845 } } }
I20260928 02:43:06.163214 31537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:06.163159 31539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:06.163465 31536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:06.161305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33601 (local address 127.9.130.62:36845)
0928 02:43:06.161511 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:06.161521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:06.161553 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:43:06.161674 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:06.161681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:06.161711 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:06.161857 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:06.162567 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:06.162571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:06.162571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:06.162575 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:06.162614 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:06.162626 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:06.162682 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:06.162936 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:43:06.162975 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":104,"threads_started":1}
I20260928 02:43:06.163468 31545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:06.164000 31545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:06.164376 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:06.165526 31545 catalog_manager.cc:1347] Generated new cluster ID: e1e8ea3bf7ae4c85bc312613aa5933ef
I20260928 02:43:06.165589 31545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:06.172881 31536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:06.168143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46259 (local address 127.9.130.62:36845)
0928 02:43:06.168181 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:06.168182 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:06.172007 (+ 3825us) server_negotiation.cc:407] Connection header received
0928 02:43:06.172067 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:06.172068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:06.172073 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:06.172126 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:06.172687 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:06.172688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:06.172688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:06.172692 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:06.172704 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:06.172708 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:06.172757 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:06.172809 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:43:06.172820 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:43:06.181667 31545 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:06.181850 31545 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:06.191501 31545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 577e70b39d2a437b9b7fa3bb9f185733: Generated new TSK 0
I20260928 02:43:06.191696 31545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:06.196405 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 31464
2026-09-28T02:43:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:06 dist-test-slave-kkrh krb5kdc[31563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:06 dist-test-slave-kkrh krb5kdc[31563](info): set up 2 sockets
Sep 28 02:43:06 dist-test-slave-kkrh krb5kdc[31563](info): commencing operation
krb5kdc: starting...
W20260928 02:43:07.231943 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:43:07 dist-test-slave-kkrh krb5kdc[31563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:43:07.346130 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:41987
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:41855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:41987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:07.425771 31576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:07.425963 31576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:07.425987 31576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:07.427050 31576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:07.427083 31576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:07.427098 31576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:07.427109 31576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:07.427121 31576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:07.428519 31576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:41855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:41987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:41987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:07.428704 31576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:07.428869 31576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:07.433589 31582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:07.433633 31584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:07.433727 31576 server_base.cc:1034] running on GCE node
W20260928 02:43:07.433586 31581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:07.433983 31576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:07.434257 31576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:07.435406 31576 hybrid_clock.cc:648] HybridClock initialized: now 1790563387435384 us; error 40 us; skew 500 ppm
Sep 28 02:43:07 dist-test-slave-kkrh krb5kdc[31563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563387, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:07.437762 31576 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:43:07.438171 31576 webserver.cc:460] Webserver started at http://127.9.130.62:37815/ using document root <none> and password file <none>
I20260928 02:43:07.438408 31576 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:07.438475 31576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:07.438625 31576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:07.439698 31576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "9e93d59e1cb14cdb8b301ea6d04dfc32"
format_stamp: "Formatted at 2026-09-28 02:43:07 on dist-test-slave-kkrh"
server_key: "19e297e22c63d80decfed5c10ed00721"
server_key_iv: "7bdbe84cd41882ae6ba92952ea336883"
server_key_version: "encryptionkey@0"
I20260928 02:43:07.440122 31576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "9e93d59e1cb14cdb8b301ea6d04dfc32"
format_stamp: "Formatted at 2026-09-28 02:43:07 on dist-test-slave-kkrh"
server_key: "19e297e22c63d80decfed5c10ed00721"
server_key_iv: "7bdbe84cd41882ae6ba92952ea336883"
server_key_version: "encryptionkey@0"
I20260928 02:43:07.441411 31576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:07.442088 31591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:07.442262 31576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:07.442365 31576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "9e93d59e1cb14cdb8b301ea6d04dfc32"
format_stamp: "Formatted at 2026-09-28 02:43:07 on dist-test-slave-kkrh"
server_key: "19e297e22c63d80decfed5c10ed00721"
server_key_iv: "7bdbe84cd41882ae6ba92952ea336883"
server_key_version: "encryptionkey@0"
I20260928 02:43:07.442437 31576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:07.493176 31576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:07.495196 31576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:07.495364 31576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:07.499377 31576 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:41987
I20260928 02:43:07.499415 31643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:41987 every 8 connection(s)
I20260928 02:43:07.499800 31576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:07.500666 31644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:07.501434 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 31576
I20260928 02:43:07.501561 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:43:07.501735 9736 external_mini_cluster.cc:1428] Setting key 33c8bdc80649f227c6d4ffeb24fa2d0b
I20260928 02:43:07.503911 31644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32: Bootstrap starting.
I20260928 02:43:07.504503 31644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:07.504812 31644 log.cc:824] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:07.505460 31644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32: No bootstrap required, opened a new log
I20260928 02:43:07.507341 31644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e93d59e1cb14cdb8b301ea6d04dfc32" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41987 } }
I20260928 02:43:07.507498 31644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:07.507566 31644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e93d59e1cb14cdb8b301ea6d04dfc32, State: Initialized, Role: FOLLOWER
I20260928 02:43:07.507737 31644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e93d59e1cb14cdb8b301ea6d04dfc32" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41987 } }
I20260928 02:43:07.507856 31644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:07.507899 31644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:07.507949 31644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:07.508636 31644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e93d59e1cb14cdb8b301ea6d04dfc32" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41987 } }
Sep 28 02:43:07 dist-test-slave-kkrh krb5kdc[31563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:43:07.508750 31644 leader_election.cc:302] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e93d59e1cb14cdb8b301ea6d04dfc32; no voters:
I20260928 02:43:07.508913 31644 leader_election.cc:288] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:07.509042 31649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:07.509174 31644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:07.509279 31649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [term 1 LEADER]: Becoming Leader. State: Replica: 9e93d59e1cb14cdb8b301ea6d04dfc32, State: Running, Role: LEADER
I20260928 02:43:07.509378 31649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e93d59e1cb14cdb8b301ea6d04dfc32" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41987 } }
I20260928 02:43:07.509730 31650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e93d59e1cb14cdb8b301ea6d04dfc32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e93d59e1cb14cdb8b301ea6d04dfc32" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41987 } } }
I20260928 02:43:07.510126 31650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:07.510350 31659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:07.510881 31659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:07.511471 31651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e93d59e1cb14cdb8b301ea6d04dfc32. Latest consensus state: current_term: 1 leader_uuid: "9e93d59e1cb14cdb8b301ea6d04dfc32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e93d59e1cb14cdb8b301ea6d04dfc32" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 41987 } } }
I20260928 02:43:07.511549 31651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:07.512537 31659 catalog_manager.cc:1347] Generated new cluster ID: 957739804db541b0abb14aa2fb64925d
I20260928 02:43:07.512626 31659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:07.513121 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:07.502254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43405 (local address 127.9.130.62:41987)
0928 02:43:07.502452 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:07.502460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:07.502487 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:43:07.502938 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:07.502945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:07.502993 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:07.503142 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:07.503972 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:07.505028 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:07.505826 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:07.506275 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:07.506403 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:07.509241 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:07.509246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:07.509246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:07.509258 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:07.511273 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:07.511693 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:07.511696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:07.511697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:07.511793 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:07.512072 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:07.512073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:07.512073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:07.512335 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:07.512475 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:07.512813 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 02:43:07.512938 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260928 02:43:07.514017 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:07.519066 31659 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:07.519537 31659 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:07.528884 31659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e93d59e1cb14cdb8b301ea6d04dfc32: Generated new TSK 0
I20260928 02:43:07.529008 31659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:07.532321 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:07.515469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55983 (local address 127.9.130.62:41987)
0928 02:43:07.515524 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:07.515526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:07.515640 (+ 114us) server_negotiation.cc:407] Connection header received
0928 02:43:07.515666 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:07.515667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:07.515678 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:07.515746 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:07.516328 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:07.517057 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:07.518041 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:07.518362 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:07.518445 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:07.528888 (+ 10443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:07.528890 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:07.528890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:07.528898 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:07.530916 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:07.531302 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:07.531303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:07.531303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:07.531377 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:07.531598 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:07.531599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:07.531600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:07.531744 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:07.531876 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:07.532116 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:43:07.532218 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:43:07.534564 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 31576
2026-09-28T02:43:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:07 dist-test-slave-kkrh krb5kdc[31675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:07 dist-test-slave-kkrh krb5kdc[31675](info): set up 2 sockets
Sep 28 02:43:07 dist-test-slave-kkrh krb5kdc[31675](info): commencing operation
krb5kdc: starting...
W20260928 02:43:08.567927 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:43:08 dist-test-slave-kkrh krb5kdc[31675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:43:08.682457 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:38043
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:33963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:38043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:08.761505 31688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:08.761729 31688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:08.761766 31688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:08.762902 31688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:08.762965 31688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:08.762991 31688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:08.763022 31688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:08.763046 31688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:08.764539 31688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:33963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:38043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:38043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:08.764803 31688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:08.765028 31688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:08.769795 31696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:08.769932 31693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:08.769829 31694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:08.769973 31688 server_base.cc:1034] running on GCE node
I20260928 02:43:08.770289 31688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:08.770514 31688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:08.771656 31688 hybrid_clock.cc:648] HybridClock initialized: now 1790563388771645 us; error 48 us; skew 500 ppm
Sep 28 02:43:08 dist-test-slave-kkrh krb5kdc[31675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563388, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:08.777549 31688 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:43:08.778012 31688 webserver.cc:460] Webserver started at http://127.9.130.62:33683/ using document root <none> and password file <none>
I20260928 02:43:08.778267 31688 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:08.778339 31688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:08.778507 31688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:08.779637 31688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "c8511b32a3894beb840ed9e1fb3aa0ea"
format_stamp: "Formatted at 2026-09-28 02:43:08 on dist-test-slave-kkrh"
server_key: "30b5e50ad30132f38d54bf1dd440e754"
server_key_iv: "a04d686e82431963fc78c63017aace21"
server_key_version: "encryptionkey@0"
I20260928 02:43:08.780045 31688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "c8511b32a3894beb840ed9e1fb3aa0ea"
format_stamp: "Formatted at 2026-09-28 02:43:08 on dist-test-slave-kkrh"
server_key: "30b5e50ad30132f38d54bf1dd440e754"
server_key_iv: "a04d686e82431963fc78c63017aace21"
server_key_version: "encryptionkey@0"
I20260928 02:43:08.781419 31688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:08.782164 31703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:08.782351 31688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:08.782459 31688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "c8511b32a3894beb840ed9e1fb3aa0ea"
format_stamp: "Formatted at 2026-09-28 02:43:08 on dist-test-slave-kkrh"
server_key: "30b5e50ad30132f38d54bf1dd440e754"
server_key_iv: "a04d686e82431963fc78c63017aace21"
server_key_version: "encryptionkey@0"
I20260928 02:43:08.782563 31688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:08.801584 31688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:08.803301 31688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:08.803458 31688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:08.807449 31688 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:38043
I20260928 02:43:08.807483 31755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:38043 every 8 connection(s)
I20260928 02:43:08.807839 31688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:08.808693 31756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:08.810879 31756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea: Bootstrap starting.
I20260928 02:43:08.811443 31756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:08.811728 31756 log.cc:824] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:08.812428 31756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea: No bootstrap required, opened a new log
I20260928 02:43:08.813959 31756 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8511b32a3894beb840ed9e1fb3aa0ea" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38043 } }
I20260928 02:43:08.814097 31756 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:08.814121 31756 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8511b32a3894beb840ed9e1fb3aa0ea, State: Initialized, Role: FOLLOWER
I20260928 02:43:08.814303 31756 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8511b32a3894beb840ed9e1fb3aa0ea" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38043 } }
I20260928 02:43:08.814384 31756 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:08.814435 31756 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:08.814482 31756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:08.815281 31756 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8511b32a3894beb840ed9e1fb3aa0ea" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38043 } }
I20260928 02:43:08.815392 31756 leader_election.cc:302] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8511b32a3894beb840ed9e1fb3aa0ea; no voters:
I20260928 02:43:08.815584 31756 leader_election.cc:288] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:08.815646 31759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:08.815783 31759 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [term 1 LEADER]: Becoming Leader. State: Replica: c8511b32a3894beb840ed9e1fb3aa0ea, State: Running, Role: LEADER
I20260928 02:43:08.815855 31756 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:08.815909 31759 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8511b32a3894beb840ed9e1fb3aa0ea" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38043 } }
I20260928 02:43:08.816311 31761 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8511b32a3894beb840ed9e1fb3aa0ea. Latest consensus state: current_term: 1 leader_uuid: "c8511b32a3894beb840ed9e1fb3aa0ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8511b32a3894beb840ed9e1fb3aa0ea" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38043 } } }
I20260928 02:43:08.816416 31761 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:08.816298 31760 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8511b32a3894beb840ed9e1fb3aa0ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8511b32a3894beb840ed9e1fb3aa0ea" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 38043 } } }
I20260928 02:43:08.816721 31760 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:08.816819 31768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:08.816907 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 31688
I20260928 02:43:08.817057 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:43:08.817251 9736 external_mini_cluster.cc:1428] Setting key 1a9fcf20f92b18d9a77e9537fe6acd7e
I20260928 02:43:08.817349 31768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:08.819039 31768 catalog_manager.cc:1347] Generated new cluster ID: 73d9dea8b7344b599c00bd3c322c9108
I20260928 02:43:08.819096 31768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:08.822707 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:08.817690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36309 (local address 127.9.130.62:38043)
0928 02:43:08.817872 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:08.817881 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:08.818507 (+ 626us) server_negotiation.cc:407] Connection header received
0928 02:43:08.818680 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:08.818687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:08.818740 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:08.818885 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:08.819126 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:08.820078 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:08.820882 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:08.821279 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:08.821402 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:08.822116 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:08.822120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:08.822120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:08.822291 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:08.822320 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:08.822331 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:08.822409 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:08.822481 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 02:43:08.822548 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260928 02:43:08.823624 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:08.830921 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:08.828181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33303 (local address 127.9.130.62:38043)
0928 02:43:08.828215 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:08.828217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:08.828337 (+ 120us) server_negotiation.cc:407] Connection header received
0928 02:43:08.828407 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:08.828408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:08.828418 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:08.828473 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:08.828590 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:08.829049 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:08.829745 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:08.829947 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:08.830029 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:08.830671 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:08.830672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:08.830672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:08.830676 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:08.830689 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:08.830694 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:08.830745 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:08.830807 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 02:43:08.830828 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:43:08.836901 31768 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:08.837404 31768 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:08.846266 31768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8511b32a3894beb840ed9e1fb3aa0ea: Generated new TSK 0
I20260928 02:43:08.846453 31768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:08.853070 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 31688
2026-09-28T02:43:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:08 dist-test-slave-kkrh krb5kdc[31787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:08 dist-test-slave-kkrh krb5kdc[31787](info): set up 2 sockets
Sep 28 02:43:08 dist-test-slave-kkrh krb5kdc[31787](info): commencing operation
krb5kdc: starting...
W20260928 02:43:09.895963 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:43:09 dist-test-slave-kkrh krb5kdc[31787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:43:10.010913 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:33117
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:44001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:33117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:10.090600 31800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:10.090795 31800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:10.090818 31800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:10.092000 31800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:10.092041 31800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:10.092054 31800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:10.092067 31800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:10.092121 31800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:10.093523 31800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:44001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:33117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:33117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:10.093760 31800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:10.093987 31800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:10.098701 31805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:10.098729 31806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:10.098821 31800 server_base.cc:1034] running on GCE node
W20260928 02:43:10.098707 31808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:10.099179 31800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:10.099484 31800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:10.100646 31800 hybrid_clock.cc:648] HybridClock initialized: now 1790563390100632 us; error 42 us; skew 500 ppm
Sep 28 02:43:10 dist-test-slave-kkrh krb5kdc[31787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563390, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:10.102963 31800 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:43:10.103375 31800 webserver.cc:460] Webserver started at http://127.9.130.62:34351/ using document root <none> and password file <none>
I20260928 02:43:10.103612 31800 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:10.103679 31800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:10.103860 31800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:10.104924 31800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "75384740c47b472190ddca55d36f678e"
format_stamp: "Formatted at 2026-09-28 02:43:10 on dist-test-slave-kkrh"
server_key: "65f6cc61741511e90984bd8dec1b69f1"
server_key_iv: "347be6d8fe27c8ff3a685ac84fc56073"
server_key_version: "encryptionkey@0"
I20260928 02:43:10.105276 31800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "75384740c47b472190ddca55d36f678e"
format_stamp: "Formatted at 2026-09-28 02:43:10 on dist-test-slave-kkrh"
server_key: "65f6cc61741511e90984bd8dec1b69f1"
server_key_iv: "347be6d8fe27c8ff3a685ac84fc56073"
server_key_version: "encryptionkey@0"
I20260928 02:43:10.106552 31800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:43:10.107268 31815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:10.107460 31800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:10.107555 31800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "75384740c47b472190ddca55d36f678e"
format_stamp: "Formatted at 2026-09-28 02:43:10 on dist-test-slave-kkrh"
server_key: "65f6cc61741511e90984bd8dec1b69f1"
server_key_iv: "347be6d8fe27c8ff3a685ac84fc56073"
server_key_version: "encryptionkey@0"
I20260928 02:43:10.107638 31800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:10.110821 31800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:10.112779 31800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:10.112933 31800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:10.116734 31800 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:33117
I20260928 02:43:10.116765 31867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:33117 every 8 connection(s)
I20260928 02:43:10.117079 31800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:10.118054 31868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:10.120244 31868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e: Bootstrap starting.
I20260928 02:43:10.120730 31868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:10.120970 31868 log.cc:824] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:10.121585 31868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e: No bootstrap required, opened a new log
I20260928 02:43:10.123106 31868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75384740c47b472190ddca55d36f678e" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33117 } }
I20260928 02:43:10.123229 31868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:10.123250 31868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75384740c47b472190ddca55d36f678e, State: Initialized, Role: FOLLOWER
I20260928 02:43:10.123384 31868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75384740c47b472190ddca55d36f678e" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33117 } }
I20260928 02:43:10.123461 31868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:10.123482 31868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:10.123509 31868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:10.124339 31868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75384740c47b472190ddca55d36f678e" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33117 } }
I20260928 02:43:10.124421 31868 leader_election.cc:302] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75384740c47b472190ddca55d36f678e; no voters:
I20260928 02:43:10.124533 31868 leader_election.cc:288] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:10.124576 31871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:10.124739 31871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [term 1 LEADER]: Becoming Leader. State: Replica: 75384740c47b472190ddca55d36f678e, State: Running, Role: LEADER
I20260928 02:43:10.124784 31868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:10.124856 31871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75384740c47b472190ddca55d36f678e" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33117 } }
I20260928 02:43:10.125305 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 31800
I20260928 02:43:10.125270 31872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75384740c47b472190ddca55d36f678e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75384740c47b472190ddca55d36f678e" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33117 } } }
I20260928 02:43:10.125370 31872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:10.125414 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:43:10.125625 9736 external_mini_cluster.cc:1428] Setting key 4fdce64b5e3f3bc323ae97a7c63143db
I20260928 02:43:10.126188 31873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75384740c47b472190ddca55d36f678e. Latest consensus state: current_term: 1 leader_uuid: "75384740c47b472190ddca55d36f678e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75384740c47b472190ddca55d36f678e" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33117 } } }
I20260928 02:43:10.126564 31873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e [sys.catalog]: This master's current role is: LEADER
W20260928 02:43:10.126896 31887 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:10.126952 31887 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:10.126994 31880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:10.127151 31880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:10.128719 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:10.126193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38359 (local address 127.9.130.62:33117)
0928 02:43:10.126802 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:10.126812 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:10.126843 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:43:10.126967 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:10.126974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:10.127008 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:10.127146 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:10.128300 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:10.128303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:10.128303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:10.128308 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:10.128347 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:10.128361 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:10.128421 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:10.128499 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 02:43:10.128537 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":68,"threads_started":1}
I20260928 02:43:10.128741 31880 catalog_manager.cc:1347] Generated new cluster ID: 481c5c645865409fa38e467445fab23a
I20260928 02:43:10.128784 31880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:10.129602 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:10.132207 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:10.131204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50747 (local address 127.9.130.62:33117)
0928 02:43:10.131264 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:10.131266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:10.131307 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:43:10.131360 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:10.131361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:10.131365 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:10.131432 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:10.132002 (+ 570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:10.132003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:10.132004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:10.132008 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:10.132021 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:10.132026 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:10.132087 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:10.132138 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 02:43:10.132151 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:43:10.146843 31880 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:10.146997 31880 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:10.152328 31880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75384740c47b472190ddca55d36f678e: Generated new TSK 0
I20260928 02:43:10.152488 31880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:10.155577 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 31800
2026-09-28T02:43:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1302 ms)
[----------] 18 tests from AuthTokenIssuingTest (23692 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:10 dist-test-slave-kkrh krb5kdc[31899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:10 dist-test-slave-kkrh krb5kdc[31899](info): set up 2 sockets
Sep 28 02:43:10 dist-test-slave-kkrh krb5kdc[31899](info): commencing operation
krb5kdc: starting...
W20260928 02:43:11.188066 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:43:11.304486 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:33023
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:46069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:33023,127.9.130.61:37323,127.9.130.60:38181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:11.386871 31912 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:43:11.387123 31912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:11.387158 31912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:11.387204 31912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:11.388527 31912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:11.388597 31912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:11.388622 31912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:11.388749 31912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:11.388784 31912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:11.390439 31912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:46069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:33023,127.9.130.61:37323,127.9.130.60:38181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:33023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:11.390748 31912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:11.390995 31912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:11.395833 31918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:11.395988 31912 server_base.cc:1034] running on GCE node
W20260928 02:43:11.395799 31920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:11.395799 31917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:11.396371 31912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:11.396653 31912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:11.397828 31912 hybrid_clock.cc:648] HybridClock initialized: now 1790563391397812 us; error 30 us; skew 500 ppm
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:11.400513 31912 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:43:11.400972 31912 webserver.cc:460] Webserver started at http://127.9.130.62:35305/ using document root <none> and password file <none>
I20260928 02:43:11.401225 31912 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:11.401301 31912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:11.401456 31912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:11.402544 31912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "abe098c82b1c4ef9b2a2a195426cff6f"
format_stamp: "Formatted at 2026-09-28 02:43:11 on dist-test-slave-kkrh"
server_key: "469f1d37c02297e168879efcab0b5f00"
server_key_iv: "d6db79104c433dc83acdb46f297c8d03"
server_key_version: "encryptionkey@0"
I20260928 02:43:11.402925 31912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "abe098c82b1c4ef9b2a2a195426cff6f"
format_stamp: "Formatted at 2026-09-28 02:43:11 on dist-test-slave-kkrh"
server_key: "469f1d37c02297e168879efcab0b5f00"
server_key_iv: "d6db79104c433dc83acdb46f297c8d03"
server_key_version: "encryptionkey@0"
I20260928 02:43:11.404421 31912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:43:11.405278 31927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:11.405462 31912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:11.405570 31912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "abe098c82b1c4ef9b2a2a195426cff6f"
format_stamp: "Formatted at 2026-09-28 02:43:11 on dist-test-slave-kkrh"
server_key: "469f1d37c02297e168879efcab0b5f00"
server_key_iv: "d6db79104c433dc83acdb46f297c8d03"
server_key_version: "encryptionkey@0"
I20260928 02:43:11.405644 31912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:11.422863 31912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:11.424855 31912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:11.425016 31912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:11.429003 31912 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:33023
I20260928 02:43:11.429119 31979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:33023 every 8 connection(s)
I20260928 02:43:11.429405 31912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:11.430413 31980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:11.431946 31980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:43:11.438920 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 31912
I20260928 02:43:11.439038 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:43:11.439230 9736 external_mini_cluster.cc:1428] Setting key 6cb5371dea08bdcb42adb4d68121752a
I20260928 02:43:11.441391 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.432614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.9.130.62:33023)
0928 02:43:11.432766 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.432770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.432830 (+ 60us) server_negotiation.cc:407] Connection header received
0928 02:43:11.433313 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.433315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.433331 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.433410 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.433900 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.434693 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.435505 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.435801 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.435894 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.437829 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.437835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.437836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.437841 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.439681 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.440096 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.440099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.440100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.440224 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.440461 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.440462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.440463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.440776 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.440879 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.441100 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.441210 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260928 02:43:11.441497 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.432561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:33023 (local address 127.0.0.1:51104)
0928 02:43:11.432717 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.432757 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:11.433190 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:11.433436 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:11.433438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:11.433665 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:11.433820 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.433824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.434771 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.434771 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.435436 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.435437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.435913 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.435914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.435967 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.436473 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:11.436481 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:11.437754 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:11.439764 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.439770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.439772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.440007 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.440313 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.440314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.440314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.440393 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.440865 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:11.440872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:11.440951 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 02:43:11.441059 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:43:11.441336 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":90,"thread_start_us":87,"threads_started":1}
I20260928 02:43:11.442351 31980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:43:11.442691 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.442583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:37323 (local address 127.0.0.1:32902)
0928 02:43:11.442629 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.442674 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:43:11.442859 31928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.130.61:37323: connect: Connection refused (error 111)
I20260928 02:43:11.443475 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.443252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:37323 (local address 127.0.0.1:32910)
0928 02:43:11.443414 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.443445 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260928 02:43:11.443602 31980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.61:37323: Network error: Client connection negotiation failed: client connection to 127.9.130.61:37323: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.9.130.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.61.keytab.
Entry for principal kudu/127.9.130.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.61.keytab.
I20260928 02:43:11.462755 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.61:37323
--webserver_interface=127.9.130.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.61.keytab
--principal=kudu/127.9.130.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:46069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:33023,127.9.130.61:37323,127.9.130.60:38181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:43:11.492861 31980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } attempt: 1
I20260928 02:43:11.493330 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.493195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:37323 (local address 127.0.0.1:32914)
0928 02:43:11.493260 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.493300 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:43:11.493674 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.493579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:37323 (local address 127.0.0.1:32924)
0928 02:43:11.493624 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.493648 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:43:11.493759 31980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.61:37323: Network error: Client connection negotiation failed: client connection to 127.9.130.61:37323: connect: Connection refused (error 111)
W20260928 02:43:11.544489 31987 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:43:11.544715 31987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:11.544744 31987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:11.544786 31987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:11.545892 31987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:11.545958 31987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:11.545984 31987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:11.546015 31987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:11.546038 31987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:11.547545 31987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:46069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:33023,127.9.130.61:37323,127.9.130.60:38181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.61.keytab
--principal=kudu/127.9.130.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.61:37323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.130.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:11.547802 31987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:11.548044 31987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:11.552634 31993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:11.552747 31992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:11.552623 31995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:11.552875 31987 server_base.cc:1034] running on GCE node
I20260928 02:43:11.553086 31987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:11.553310 31987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:11.554445 31987 hybrid_clock.cc:648] HybridClock initialized: now 1790563391554433 us; error 29 us; skew 500 ppm
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:11.557039 31987 init.cc:377] Logged in from keytab as kudu/127.9.130.61@KRBTEST.COM (short username kudu)
I20260928 02:43:11.557510 31987 webserver.cc:460] Webserver started at http://127.9.130.61:44761/ using document root <none> and password file <none>
I20260928 02:43:11.557739 31987 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:11.557802 31987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:11.557948 31987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:11.559144 31987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/data/instance:
uuid: "91702050d2044e0d846c92fde5ddbb8a"
format_stamp: "Formatted at 2026-09-28 02:43:11 on dist-test-slave-kkrh"
server_key: "b4d80bfae96efc2cf9406c1ef7955543"
server_key_iv: "63c8349ca30125aac24b6e56f67b960d"
server_key_version: "encryptionkey@0"
I20260928 02:43:11.559523 31987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/wal/instance:
uuid: "91702050d2044e0d846c92fde5ddbb8a"
format_stamp: "Formatted at 2026-09-28 02:43:11 on dist-test-slave-kkrh"
server_key: "b4d80bfae96efc2cf9406c1ef7955543"
server_key_iv: "63c8349ca30125aac24b6e56f67b960d"
server_key_version: "encryptionkey@0"
I20260928 02:43:11.560904 31987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:43:11.561616 32002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:11.561816 31987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:11.561911 31987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/wal
uuid: "91702050d2044e0d846c92fde5ddbb8a"
format_stamp: "Formatted at 2026-09-28 02:43:11 on dist-test-slave-kkrh"
server_key: "b4d80bfae96efc2cf9406c1ef7955543"
server_key_iv: "63c8349ca30125aac24b6e56f67b960d"
server_key_version: "encryptionkey@0"
I20260928 02:43:11.561909 31980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } attempt: 2
I20260928 02:43:11.561992 31987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:11.562281 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.562155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:37323 (local address 127.0.0.1:32936)
0928 02:43:11.562217 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.562249 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 02:43:11.562573 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.562479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:37323 (local address 127.0.0.1:32946)
0928 02:43:11.562522 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.562546 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 02:43:11.562729 31980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.61:37323: Network error: Client connection negotiation failed: client connection to 127.9.130.61:37323: connect: Connection refused (error 111)
I20260928 02:43:11.590242 31987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:11.592617 31987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:11.592794 31987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:11.597162 31987 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.61:37323
I20260928 02:43:11.597208 32054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.61:37323 every 8 connection(s)
I20260928 02:43:11.597589 31987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/data/info.pb
I20260928 02:43:11.598497 32055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:11.600134 32055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.61@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:43:11.607591 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 31987
I20260928 02:43:11.607702 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-1/wal/instance
I20260928 02:43:11.607959 9736 external_mini_cluster.cc:1428] Setting key 9ef221d0c344d606d36a4634ddbf7f69
I20260928 02:43:11.608749 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.600841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.9.130.62:33023)
0928 02:43:11.600909 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.600912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.601149 (+ 237us) server_negotiation.cc:407] Connection header received
0928 02:43:11.601602 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.601603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.601616 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.601677 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.602218 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.602692 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.603609 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.603801 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.603946 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.605921 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.605922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.605923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.605929 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.607123 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.607501 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.607502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.607502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.607570 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.608003 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.608005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.608005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.608137 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.608213 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.608535 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.608619 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:43:11.608840 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.600839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:33023 (local address 127.0.0.1:51114)
0928 02:43:11.601037 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.601069 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:11.601514 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:11.601730 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:11.601732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:11.601954 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:11.602139 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.602142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.602770 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.602771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.603538 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.603540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.603978 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.603979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.604085 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.604575 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:11.604582 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:11.605842 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:11.607200 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.607219 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.607220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.607423 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.607730 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.607731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.607731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.607922 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.608203 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:11.608224 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:11.608376 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 02:43:11.608496 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:43:11.608640 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":121,"threads_started":1}
I20260928 02:43:11.609580 32055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.61@KRBTEST.COM for kudu/127.9.130.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.130.60@KRBTEST.COM; defaulting to no policy
I20260928 02:43:11.617393 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.609833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:37323 (local address 127.0.0.1:32948)
0928 02:43:11.609882 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.609892 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:11.609927 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:11.610167 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:11.610168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:11.610306 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:11.610344 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.610345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.611596 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.611597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.612190 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.612192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.612894 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.612895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.612948 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.613360 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:11.613364 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:11.614451 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:11.616392 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.616393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.616393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.616546 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.616739 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.616740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.616740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.616787 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.617176 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:11.617177 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:11.617212 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:43:11.617265 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 02:43:11.617335 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:43:11.617558 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.609897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.9.130.61:37323)
0928 02:43:11.610038 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.610049 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.610063 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:43:11.610079 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.610080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.610089 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.610168 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.610428 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.611499 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.612267 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.612809 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.612891 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.614531 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.614538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.614538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.614546 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.616324 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.616615 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.616617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.616617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.616666 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.616953 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.616955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.616955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.617107 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.617196 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.617295 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.617389 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260928 02:43:11.618150 32055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.60@KRBTEST.COM" created.
I20260928 02:43:11.618624 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.618442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:38181 (local address 127.0.0.1:34278)
0928 02:43:11.618505 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.618599 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 02:43:11.618796 32006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
I20260928 02:43:11.619203 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.619104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:38181 (local address 127.0.0.1:34284)
0928 02:43:11.619155 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.619186 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 02:43:11.619351 32055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:38181: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.60.keytab.
Entry for principal kudu/127.9.130.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.60.keytab.
I20260928 02:43:11.632745 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.60:38181
--webserver_interface=127.9.130.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.60.keytab
--principal=kudu/127.9.130.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:46069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:33023,127.9.130.61:37323,127.9.130.60:38181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:43:11.653872 31980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } attempt: 3
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for kudu/127.9.130.61@KRBTEST.COM
I20260928 02:43:11.661175 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.654210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:37323 (local address 127.0.0.1:32954)
0928 02:43:11.654273 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.654282 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:11.654345 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:11.654498 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:11.654499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:11.654661 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:11.654723 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.654725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.655416 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.655417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.656090 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.656094 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.656473 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.656474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.656555 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.657007 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:11.657012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:11.658556 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:11.660299 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.660300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.660300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.660457 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.660650 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.660651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.660651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.660700 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.660951 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:11.660952 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:11.660988 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:43:11.661038 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:43:11.661107 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 02:43:11.661224 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.654278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.9.130.61:37323)
0928 02:43:11.654340 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.654342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.654354 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:43:11.654406 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.654407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.654426 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.654492 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.654822 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.655319 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.656174 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.656395 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.656475 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.658637 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.658638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.658639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.658647 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.660207 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.660525 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.660526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.660526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.660586 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.660758 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.660759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.660760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.660888 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.660956 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.661060 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.661136 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:43:11.661585 31980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:43:11.661906 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.661813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:38181 (local address 127.0.0.1:34290)
0928 02:43:11.661856 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.661886 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:43:11.662204 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.662154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:38181 (local address 127.0.0.1:34296)
0928 02:43:11.662174 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.662193 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 02:43:11.662310 31980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:38181: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
I20260928 02:43:11.668565 32055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } attempt: 1
I20260928 02:43:11.668937 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.668809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:38181 (local address 127.0.0.1:34308)
0928 02:43:11.668863 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.668901 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 02:43:11.669298 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.669174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:38181 (local address 127.0.0.1:34314)
0928 02:43:11.669227 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.669268 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:43:11.669421 32055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:38181: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
I20260928 02:43:11.693481 31980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } attempt: 1
I20260928 02:43:11.693985 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.693839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:38181 (local address 127.0.0.1:34328)
0928 02:43:11.693905 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.693951 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:43:11.694375 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.694277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:38181 (local address 127.0.0.1:34340)
0928 02:43:11.694326 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.694354 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:43:11.694504 31980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:38181: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
W20260928 02:43:11.713662 32062 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:43:11.713903 32062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:11.713943 32062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:11.713989 32062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:11.715112 32062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:11.715176 32062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:11.715201 32062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:11.715231 32062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:11.715257 32062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:11.716768 32062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:46069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:33023,127.9.130.61:37323,127.9.130.60:38181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.60.keytab
--principal=kudu/127.9.130.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.60:38181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.130.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:11.717027 32062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:11.717245 32062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:11.722013 32070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:11.722069 32068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:11.722057 32062 server_base.cc:1034] running on GCE node
W20260928 02:43:11.722026 32067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:11.722414 32062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:11.722687 32062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:11.723865 32062 hybrid_clock.cc:648] HybridClock initialized: now 1790563391723866 us; error 40 us; skew 500 ppm
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:11.726204 32062 init.cc:377] Logged in from keytab as kudu/127.9.130.60@KRBTEST.COM (short username kudu)
I20260928 02:43:11.726586 32062 webserver.cc:460] Webserver started at http://127.9.130.60:33905/ using document root <none> and password file <none>
I20260928 02:43:11.726753 32062 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:11.726796 32062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:11.726879 32062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:11.727864 32062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/data/instance:
uuid: "b19a4cddecce470bae977b9729661717"
format_stamp: "Formatted at 2026-09-28 02:43:11 on dist-test-slave-kkrh"
server_key: "6e469d1321bad7abce8f4ab604f348d2"
server_key_iv: "3ae488df0ef085b21c106290d4194d3f"
server_key_version: "encryptionkey@0"
I20260928 02:43:11.728178 32062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/wal/instance:
uuid: "b19a4cddecce470bae977b9729661717"
format_stamp: "Formatted at 2026-09-28 02:43:11 on dist-test-slave-kkrh"
server_key: "6e469d1321bad7abce8f4ab604f348d2"
server_key_iv: "3ae488df0ef085b21c106290d4194d3f"
server_key_version: "encryptionkey@0"
I20260928 02:43:11.729379 32062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:11.730139 32077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:11.730338 32062 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:11.730450 32062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/wal
uuid: "b19a4cddecce470bae977b9729661717"
format_stamp: "Formatted at 2026-09-28 02:43:11 on dist-test-slave-kkrh"
server_key: "6e469d1321bad7abce8f4ab604f348d2"
server_key_iv: "3ae488df0ef085b21c106290d4194d3f"
server_key_version: "encryptionkey@0"
I20260928 02:43:11.730530 32062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:11.737588 32055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } attempt: 2
I20260928 02:43:11.737980 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.737857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:38181 (local address 127.0.0.1:34348)
0928 02:43:11.737916 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.737952 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:43:11.738303 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.738203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:38181 (local address 127.0.0.1:34352)
0928 02:43:11.738250 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.738277 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 02:43:11.738396 32055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:38181: Network error: Client connection negotiation failed: client connection to 127.9.130.60:38181: connect: Connection refused (error 111)
I20260928 02:43:11.742728 32062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:11.744851 32062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:11.744998 32062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:11.748665 32062 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.60:38181
I20260928 02:43:11.748704 32129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.60:38181 every 8 connection(s)
I20260928 02:43:11.749042 32062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/data/info.pb
I20260928 02:43:11.749957 32130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:11.751335 32130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.60@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:43:11.756814 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 32062
I20260928 02:43:11.756914 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563127283700-9736-0/minicluster-data/master-2/wal/instance
I20260928 02:43:11.757125 9736 external_mini_cluster.cc:1428] Setting key 446cb7390b90fd81e4a5609c2ed962f8
I20260928 02:43:11.760174 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.751948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51130 (local address 127.9.130.62:33023)
0928 02:43:11.752006 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.752008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.752219 (+ 211us) server_negotiation.cc:407] Connection header received
0928 02:43:11.752718 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.752719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.752732 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.752786 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.753360 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.753844 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.754842 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.755033 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.755121 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.757186 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.757188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.757188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.757196 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.758803 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.759236 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.759237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.759237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.759296 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.759536 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.759536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.759536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.759631 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.759677 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.760004 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.760083 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:43:11.760234 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.751953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:33023 (local address 127.0.0.1:51130)
0928 02:43:11.752112 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.752144 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:11.752605 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:11.752861 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:11.752865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:11.753081 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:11.753261 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.753265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.753963 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.753964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.754739 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.754741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.755151 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.755153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.755271 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.755682 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:11.755689 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:11.757099 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:11.758878 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.758882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.758884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.759159 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.759356 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.759356 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.759357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.759475 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.759692 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:11.759695 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:11.759868 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 02:43:11.759963 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:43:11.760072 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260928 02:43:11.760728 32130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.60@KRBTEST.COM
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.60@KRBTEST.COM for kudu/127.9.130.61@KRBTEST.COM
I20260928 02:43:11.766148 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.757520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.9.130.60:38181)
0928 02:43:11.757715 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.757719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.757759 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:43:11.757817 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.757820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.757839 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.757896 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.758647 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.759639 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.760679 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.761131 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.761221 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.763352 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.763359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.763359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.763366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.764986 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.765353 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.765355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.765356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.765415 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.765613 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.765614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.765614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.765746 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.765880 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.765964 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.766037 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260928 02:43:11.767371 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.761045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.9.130.61:37323)
0928 02:43:11.761097 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.761099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.761107 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:43:11.761129 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.761130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.761139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.761213 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.761465 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.761913 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.762554 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.762796 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.762867 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.764828 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.764829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.764829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.764837 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.766316 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.766643 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.766644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.766644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.766696 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.766902 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.766903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.766904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.767041 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.767100 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.767261 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.767315 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:43:11.767450 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.760968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:37323 (local address 127.0.0.1:32966)
0928 02:43:11.761012 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.761022 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:11.761057 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:11.761218 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:11.761218 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:11.761350 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:11.761390 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.761391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.761989 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.761990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.762466 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.762467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.762863 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.762864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.762928 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.763310 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:11.763314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:11.764683 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:11.766400 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.766401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.766402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.766562 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.766749 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.766749 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.766749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.766804 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.767111 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:11.767112 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:11.767154 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:43:11.767240 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:43:11.767338 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 02:43:11.768283 32130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:43:11.769649 31980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } attempt: 2
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.60@KRBTEST.COM for kudu/127.9.130.60@KRBTEST.COM
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for kudu/127.9.130.60@KRBTEST.COM
I20260928 02:43:11.775189 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.766842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.9.130.62:33023)
0928 02:43:11.766899 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.766901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.766908 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:43:11.766954 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.766955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.766964 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.767023 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.767642 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.768198 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.769085 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.769282 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.769357 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.772067 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.772068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.772068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.772075 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.773864 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.774246 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.774247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.774248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.774320 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.774551 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.774552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.774552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.774682 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.774829 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.775029 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.775100 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:43:11.775439 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.768529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:38181 (local address 127.0.0.1:34376)
0928 02:43:11.768615 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.768624 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:11.768672 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:11.768783 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:11.768784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:11.768921 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:11.768948 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.768949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.769503 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.769504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.770368 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.770371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.770801 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.770802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.770849 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.771213 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:11.771217 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:11.772494 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:11.774485 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.774486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.774486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.774686 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.774875 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.774875 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.774876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.774925 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.775160 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:11.775160 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:11.775196 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:43:11.775257 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:43:11.775350 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:43:11.776204 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.768576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.9.130.60:38181)
0928 02:43:11.768615 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.768617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.768660 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:43:11.768723 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.768723 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.768730 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.768768 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.769010 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.769435 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.770451 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.770726 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.770803 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.772575 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.772575 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.772576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.772582 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.774401 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.774756 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.774757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.774757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.774807 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.774995 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.774995 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.774996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.775096 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.775166 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.775622 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.776097 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:43:11.778131 32130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717: Bootstrap starting.
I20260928 02:43:11.778154 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.769960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:38181 (local address 127.0.0.1:34382)
0928 02:43:11.770006 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.770017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:11.770065 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:11.770278 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:11.770279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:11.770451 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:11.770486 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.770488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.771419 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.771420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.772208 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.772210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.772916 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.772917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.772974 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.773386 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:11.773392 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:11.774966 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:11.777137 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.777138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.777138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.777297 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.777521 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.777522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.777522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.777577 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.777888 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:11.777889 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:11.777931 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:43:11.777978 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:43:11.778062 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:43:11.778509 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.770007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34382 (local address 127.9.130.60:38181)
0928 02:43:11.770151 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.770152 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.770159 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:43:11.770177 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.770178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.770192 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.770269 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.770566 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.771332 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.772514 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.772820 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.772908 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.775393 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.775394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.775394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.775401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.777064 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.777373 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.777374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.777375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.777427 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.777650 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.777651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.777651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.777802 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.778268 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.778309 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.778381 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":103,"threads_started":1}
I20260928 02:43:11.778908 32130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:11.779181 32130 log.cc:824] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:11.779999 32130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717: No bootstrap required, opened a new log
I20260928 02:43:11.780164 31980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f: Bootstrap starting.
I20260928 02:43:11.780818 31980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f: Neither blocks nor log segments found. Creating new log.
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.61@KRBTEST.COM
I20260928 02:43:11.781111 31980 log.cc:824] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:11.781929 32130 raft_consensus.cc:348] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } }
I20260928 02:43:11.782073 32130 raft_consensus.cc:729] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19a4cddecce470bae977b9729661717, State: Initialized, Role: FOLLOWER
I20260928 02:43:11.782161 32130 consensus_queue.cc:260] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } }
I20260928 02:43:11.782439 32130 sys_catalog.cc:564] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:11.782419 32138 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } } }
I20260928 02:43:11.782569 32138 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:43:11.784252 32149 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:11.784298 32149 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:11.784338 31980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f: No bootstrap required, opened a new log
I20260928 02:43:11.784541 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.776117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.9.130.61:37323)
0928 02:43:11.776156 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.776158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.776166 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:43:11.776189 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.776190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.776199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.776261 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.777022 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.777650 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.778652 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.778864 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.778940 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.781369 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.781370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.781370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.781377 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.783064 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.783392 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.783393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.783394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.783446 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.783722 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.783723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.783724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.783884 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.784036 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.784383 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.784460 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:43:11.786037 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:11.786448 31980 raft_consensus.cc:348] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } }
I20260928 02:43:11.786576 31980 raft_consensus.cc:729] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe098c82b1c4ef9b2a2a195426cff6f, State: Initialized, Role: FOLLOWER
I20260928 02:43:11.786641 31980 consensus_queue.cc:260] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } }
I20260928 02:43:11.786937 31980 sys_catalog.cc:564] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:11.786907 32151 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } } }
I20260928 02:43:11.787015 32151 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:11.787076 31954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "abe098c82b1c4ef9b2a2a195426cff6f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51140
I20260928 02:43:11.787191 31954 raft_consensus.cc:482] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:43:11.787227 31954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:11.788112 31954 raft_consensus.cc:504] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } }
I20260928 02:43:11.788357 31954 leader_election.cc:288] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [CANDIDATE]: Term 1 election: Requested vote from peers 91702050d2044e0d846c92fde5ddbb8a (127.9.130.61:37323), b19a4cddecce470bae977b9729661717 (127.9.130.60:38181)
I20260928 02:43:11.788640 32104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b19a4cddecce470bae977b9729661717"
I20260928 02:43:11.788802 32104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717 [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:43:11.788945 32162 catalog_manager.cc:1558] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:11.789034 32162 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:11.789624 32104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abe098c82b1c4ef9b2a2a195426cff6f in term 1.
I20260928 02:43:11.789839 31931 leader_election.cc:302] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abe098c82b1c4ef9b2a2a195426cff6f, b19a4cddecce470bae977b9729661717; no voters:
I20260928 02:43:11.789971 32151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:11.790145 32151 raft_consensus.cc:686] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [term 1 LEADER]: Becoming Leader. State: Replica: abe098c82b1c4ef9b2a2a195426cff6f, State: Running, Role: LEADER
I20260928 02:43:11.790239 32151 consensus_queue.cc:237] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } }
I20260928 02:43:11.790652 32164 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe098c82b1c4ef9b2a2a195426cff6f. Latest consensus state: current_term: 1 leader_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } } }
I20260928 02:43:11.790777 32164 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:11.793298 32166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:11.793480 32166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:11.791415 31912 master_runner.cc:186] Error getting master registration for 127.9.130.61:37323: OK, instance_id { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" instance_seqno: 1790563391595951 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:43:11.793629 31912 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:43:11.794875 32104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717 [term 1 FOLLOWER]: Refusing update from remote peer abe098c82b1c4ef9b2a2a195426cff6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:11.795344 32164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:11.793308 32062 master_runner.cc:186] Error getting master registration for 127.9.130.61:37323: OK, instance_id { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" instance_seqno: 1790563391595951 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:43:11.796120 32062 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:43:11.796726 32138 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe098c82b1c4ef9b2a2a195426cff6f. Latest consensus state: current_term: 1 leader_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } } }
I20260928 02:43:11.796823 32138 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:11.797923 32164 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } } }
I20260928 02:43:11.797999 32164 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:11.798367 32172 mvcc.cc:205] Tried to move back new op lower bound from 7334147652789772288 to 7334147652773715968. Current Snapshot: MvccSnapshot[applied={T|T < 7334147652789772288}]
I20260928 02:43:11.798456 32138 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } } }
I20260928 02:43:11.798522 32138 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:11.798780 32166 catalog_manager.cc:1347] Generated new cluster ID: 44373c9721d74eb1b3d1f15e39db2fcc
I20260928 02:43:11.798945 32166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:11.821226 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.800105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.9.130.62:33023)
0928 02:43:11.800149 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.800151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.812111 (+ 11960us) server_negotiation.cc:407] Connection header received
0928 02:43:11.812201 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.812202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.812212 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.812243 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.813026 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.813773 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.814800 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.815091 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.815131 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.817021 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.817023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.817023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.817030 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.818979 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.819933 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.819934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.819934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.819994 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.820663 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.820664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.820664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.820791 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.820869 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.821051 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.821123 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:43:11.822454 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.800026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.9.130.61:37323)
0928 02:43:11.800063 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.800065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.811956 (+ 11891us) server_negotiation.cc:407] Connection header received
0928 02:43:11.812030 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.812031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.812057 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.812118 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.813384 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.814141 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.815910 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.816212 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.816253 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.818462 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.818463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.818464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.818470 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.820161 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.820786 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.820787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.820788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.820840 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.821138 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.821139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.821139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.821269 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.821334 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.822283 (+ 949us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.822364 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:43:11.829519 32055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } attempt: 3
I20260928 02:43:11.832576 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.800392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.9.130.60:38181)
0928 02:43:11.800423 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.800425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.819712 (+ 19287us) server_negotiation.cc:407] Connection header received
0928 02:43:11.819829 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.819830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.819842 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.819874 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.822842 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.823591 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.826054 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.826374 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.826416 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.828807 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.828808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.828808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.828816 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.830786 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.831432 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.831433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.831434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.831504 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.831847 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.831848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.831848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.831972 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.832054 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.832393 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.832482 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 28 02:43:11 dist-test-slave-kkrh krb5kdc[31899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563391, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.61@KRBTEST.COM for kudu/127.9.130.60@KRBTEST.COM
I20260928 02:43:11.836946 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.830039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.9.130.60:38181)
0928 02:43:11.830076 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.830078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.830085 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:43:11.830101 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.830102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.830110 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.830154 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.830458 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.831129 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.832049 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.832322 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.832369 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.834274 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:11.834275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:11.834275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:11.834281 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:11.835861 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.836236 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.836237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.836237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.836287 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:11.836471 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:11.836472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:11.836472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:11.836575 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:11.836628 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:11.836800 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 02:43:11.836865 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:43:11.836951 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:11.829788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:38181 (local address 127.0.0.1:34390)
0928 02:43:11.829908 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:11.829918 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:11.829958 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:11.830187 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:11.830187 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:11.830332 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:11.830387 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.830389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.831200 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.831201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.831972 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:11.831973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:11.832409 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:11.832409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:11.832455 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:11.832855 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:11.832860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:11.834188 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:11.835938 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.835939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.835940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.836163 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.836344 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:11.836344 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:11.836345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:11.836415 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:11.836638 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:11.836638 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:11.836692 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 02:43:11.836771 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:43:11.836857 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 02:43:11.838351 32055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a: Bootstrap starting.
I20260928 02:43:11.839023 32055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:11.839300 32055 log.cc:824] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:11.848515 32055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a: No bootstrap required, opened a new log
I20260928 02:43:11.850560 32055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } }
I20260928 02:43:11.850692 32055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91702050d2044e0d846c92fde5ddbb8a, State: Initialized, Role: FOLLOWER
I20260928 02:43:11.850775 32055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } }
I20260928 02:43:11.851083 32180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } } }
I20260928 02:43:11.851186 32180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:11.851917 32029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "91702050d2044e0d846c92fde5ddbb8a"
I20260928 02:43:11.852041 32029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:11.852617 32166 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:11.852896 32029 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abe098c82b1c4ef9b2a2a195426cff6f in term 1.
I20260928 02:43:11.853247 32028 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a [term 1 FOLLOWER]: Refusing update from remote peer abe098c82b1c4ef9b2a2a195426cff6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:11.853531 32166 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:11.853523 32164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:11.851102 32055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:43:11.857733 32194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:11.857791 32194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:11.857901 32183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } } }
I20260928 02:43:11.857981 32183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:11.856879 32180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe098c82b1c4ef9b2a2a195426cff6f. Latest consensus state: current_term: 1 leader_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } } }
I20260928 02:43:11.859155 32180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:11.859086 32181 mvcc.cc:205] Tried to move back new op lower bound from 7334147653011034112 to 7334147652773715968. Current Snapshot: MvccSnapshot[applied={T|T < 7334147652789772288}]
I20260928 02:43:11.862185 32164 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe098c82b1c4ef9b2a2a195426cff6f" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 33023 } } peers { permanent_uuid: "91702050d2044e0d846c92fde5ddbb8a" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 37323 } } peers { permanent_uuid: "b19a4cddecce470bae977b9729661717" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 38181 } } }
I20260928 02:43:11.862286 32164 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:11.863737 32166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abe098c82b1c4ef9b2a2a195426cff6f: Generated new TSK 0
I20260928 02:43:11.863926 32166 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:43:11.888075 32133 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:11.886799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.9.130.60:38181)
0928 02:43:11.886838 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.886841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.887422 (+ 581us) server_negotiation.cc:407] Connection header received
0928 02:43:11.887526 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.887527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.887538 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.887568 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.888014 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34398: BlockingRecv error: recv got EOF from 127.0.0.1:34398 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 02:43:11.888922 31983 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:11.887148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.9.130.62:33023)
0928 02:43:11.887186 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.887188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.888439 (+ 1251us) server_negotiation.cc:407] Connection header received
0928 02:43:11.888531 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.888532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.888541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.888569 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.888865 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51152: BlockingRecv error: recv got EOF from 127.0.0.1:51152 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 02:43:11.890138 32059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:11.889407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32984 (local address 127.9.130.61:37323)
0928 02:43:11.889444 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.889446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.889571 (+ 125us) server_negotiation.cc:407] Connection header received
0928 02:43:11.889670 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.889672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.889681 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.889717 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.890070 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32984: BlockingRecv error: recv got EOF from 127.0.0.1:32984 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 02:43:11.892427 32059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:11.891697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.9.130.61:37323)
0928 02:43:11.891729 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.891731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.891939 (+ 208us) server_negotiation.cc:407] Connection header received
0928 02:43:11.892041 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.892042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.892059 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.892087 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.892385 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33000: BlockingRecv error: recv got EOF from 127.0.0.1:33000 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 02:43:11.895453 32059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:11.894721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.9.130.61:37323)
0928 02:43:11.894763 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.894765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.894967 (+ 202us) server_negotiation.cc:407] Connection header received
0928 02:43:11.895062 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.895063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.895074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.895103 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.895413 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33010: BlockingRecv error: recv got EOF from 127.0.0.1:33010 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 02:43:11.899250 32059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:11.898579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.9.130.61:37323)
0928 02:43:11.898609 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.898611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.898805 (+ 194us) server_negotiation.cc:407] Connection header received
0928 02:43:11.898892 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.898892 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.898901 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.898931 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.899211 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33020: BlockingRecv error: recv got EOF from 127.0.0.1:33020 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 02:43:11.905202 32059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:11.904469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33032 (local address 127.9.130.61:37323)
0928 02:43:11.904501 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.904503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.904712 (+ 209us) server_negotiation.cc:407] Connection header received
0928 02:43:11.904820 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.904821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.904830 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.904859 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.905162 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33032: BlockingRecv error: recv got EOF from 127.0.0.1:33032 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 02:43:11.915197 32059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:11.914404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.9.130.61:37323)
0928 02:43:11.914439 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.914441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.914662 (+ 221us) server_negotiation.cc:407] Connection header received
0928 02:43:11.914770 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.914771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.914781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.914810 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.915141 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33034: BlockingRecv error: recv got EOF from 127.0.0.1:33034 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 02:43:11.933590 32059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:11.932639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.9.130.61:37323)
0928 02:43:11.932679 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.932682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.932942 (+ 260us) server_negotiation.cc:407] Connection header received
0928 02:43:11.933043 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.933043 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.933054 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.933084 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.933524 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33038: BlockingRecv error: recv got EOF from 127.0.0.1:33038 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 02:43:11.967851 32059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:11.967066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.9.130.61:37323)
0928 02:43:11.967109 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:11.967112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:11.967270 (+ 158us) server_negotiation.cc:407] Connection header received
0928 02:43:11.967349 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:11.967350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:11.967362 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:11.967425 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:11.967758 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33044: BlockingRecv error: recv got EOF from 127.0.0.1:33044 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 02:43:12.033806 32059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:12.033000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.9.130.61:37323)
0928 02:43:12.033085 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:12.033087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:12.033152 (+ 65us) server_negotiation.cc:407] Connection header received
0928 02:43:12.033232 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:12.033233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:12.033243 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:12.033295 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:12.033730 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33052: BlockingRecv error: recv got EOF from 127.0.0.1:33052 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 02:43:12.163690 32059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:12.163002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.9.130.61:37323)
0928 02:43:12.163063 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:12.163066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:12.163099 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:43:12.163203 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:12.163204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:12.163216 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:12.163292 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:12.163622 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33068: BlockingRecv error: recv got EOF from 127.0.0.1:33068 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 02:43:12.421700 32059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:12.420949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.9.130.61:37323)
0928 02:43:12.421027 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:12.421030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:12.421100 (+ 70us) server_negotiation.cc:407] Connection header received
0928 02:43:12.421210 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:12.421211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:12.421223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:12.421295 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:12.421634 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33080: BlockingRecv error: recv got EOF from 127.0.0.1:33080 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:43:12.784767 32149 catalog_manager.cc:1259] Loaded cluster ID: 44373c9721d74eb1b3d1f15e39db2fcc
I20260928 02:43:12.784843 32149 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717: loading cluster ID for follower catalog manager: success
I20260928 02:43:12.785599 32149 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717: acquiring CA information for follower catalog manager: success
I20260928 02:43:12.785837 32149 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b19a4cddecce470bae977b9729661717: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:43:12.858232 32194 catalog_manager.cc:1259] Loaded cluster ID: 44373c9721d74eb1b3d1f15e39db2fcc
I20260928 02:43:12.858320 32194 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a: loading cluster ID for follower catalog manager: success
I20260928 02:43:12.859110 32194 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a: acquiring CA information for follower catalog manager: success
I20260928 02:43:12.859364 32194 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91702050d2044e0d846c92fde5ddbb8a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:43:12.937331 32263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:12.934826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.9.130.61:37323)
0928 02:43:12.934945 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:12.934947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:12.934958 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:43:12.935028 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:12.935029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:12.935077 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:12.935152 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:12.935293 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:12.935806 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:12.936599 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:12.936801 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:12.936849 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:12.937001 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:12.937105 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:12.937164 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:12.937200 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 02:43:12.937216 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260928 02:43:12.941740 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:12.939224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.9.130.60:38181)
0928 02:43:12.939343 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:12.939345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:12.939376 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:43:12.939442 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:12.939443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:12.939474 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:12.939538 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:12.939672 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:12.940196 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:12.940914 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:12.941096 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:12.941144 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:12.941352 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:12.941446 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:12.941510 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:12.941591 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 02:43:12.941611 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260928 02:43:12.946380 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:12.943651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.9.130.60:38181)
0928 02:43:12.943686 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:12.943688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:12.943787 (+ 99us) server_negotiation.cc:407] Connection header received
0928 02:43:12.943898 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:12.943899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:12.943912 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:12.943973 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:12.944160 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:12.944626 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:12.945560 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:12.945835 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:12.945920 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:12.946105 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:12.946136 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:12.946166 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:12.946270 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 02:43:12.946295 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:43:12.946489 32263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:12.943512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.9.130.61:37323)
0928 02:43:12.943554 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:12.943556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:12.943794 (+ 238us) server_negotiation.cc:407] Connection header received
0928 02:43:12.943912 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:12.943913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:12.943925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:12.943972 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:12.944145 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:12.944873 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:12.945740 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:12.945993 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:12.946071 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:12.946225 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:12.946258 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:12.946311 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:12.946399 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 02:43:12.946420 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:43:12.946980 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:12.943533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51162 (local address 127.9.130.62:33023)
0928 02:43:12.943652 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:12.943654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:12.943893 (+ 239us) server_negotiation.cc:407] Connection header received
0928 02:43:12.943942 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:12.943943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:12.943992 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:12.944067 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:12.944202 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:12.944957 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:12.945899 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:12.946218 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:12.946284 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:12.946535 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:12.946646 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:12.946707 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:12.946877 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 02:43:12.946904 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260928 02:43:12.948884 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 31912
I20260928 02:43:12.954540 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 31987
I20260928 02:43:12.958967 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 32062
2026-09-28T02:43:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2804 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:13 dist-test-slave-kkrh krb5kdc[32280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:13 dist-test-slave-kkrh krb5kdc[32280](info): set up 2 sockets
Sep 28 02:43:13 dist-test-slave-kkrh krb5kdc[32280](info): commencing operation
krb5kdc: starting...
W20260928 02:43:13.992156 9736 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/test-user.keytab.
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.130.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
Entry for principal kudu/127.9.130.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab.
I20260928 02:43:14.110289 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:35991
--webserver_interface=127.9.130.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:35991,127.9.130.61:38775,127.9.130.60:35755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:14.190815 32293 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:43:14.191021 32293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:14.191038 32293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:14.191057 32293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:14.192214 32293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:14.192255 32293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:14.192268 32293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:14.192281 32293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:14.192293 32293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:14.193754 32293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:35991,127.9.130.61:38775,127.9.130.60:35755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.62.keytab
--principal=kudu/127.9.130.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.62:35991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.130.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:14.194005 32293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:14.194235 32293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:14.199225 32298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:14.199234 32301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:14.199235 32299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:14.199517 32293 server_base.cc:1034] running on GCE node
I20260928 02:43:14.199753 32293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:14.200058 32293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:14.201228 32293 hybrid_clock.cc:648] HybridClock initialized: now 1790563394201212 us; error 29 us; skew 500 ppm
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:14.206387 32293 init.cc:377] Logged in from keytab as kudu/127.9.130.62@KRBTEST.COM (short username kudu)
I20260928 02:43:14.206832 32293 webserver.cc:460] Webserver started at http://127.9.130.62:37309/ using document root <none> and password file <none>
I20260928 02:43:14.207052 32293 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:14.207103 32293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:14.207252 32293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:14.208352 32293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/data/instance:
uuid: "7ef91b599c2f45efb3279c13bf4d3441"
format_stamp: "Formatted at 2026-09-28 02:43:14 on dist-test-slave-kkrh"
server_key: "c06a94b566feaccc85ef4eac1030a5ed"
server_key_iv: "bcf042ba1b302cbd5c80759caabf1200"
server_key_version: "encryptionkey@0"
I20260928 02:43:14.208714 32293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/wal/instance:
uuid: "7ef91b599c2f45efb3279c13bf4d3441"
format_stamp: "Formatted at 2026-09-28 02:43:14 on dist-test-slave-kkrh"
server_key: "c06a94b566feaccc85ef4eac1030a5ed"
server_key_iv: "bcf042ba1b302cbd5c80759caabf1200"
server_key_version: "encryptionkey@0"
I20260928 02:43:14.209995 32293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:14.210726 32308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:14.210893 32293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:14.210997 32293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/wal
uuid: "7ef91b599c2f45efb3279c13bf4d3441"
format_stamp: "Formatted at 2026-09-28 02:43:14 on dist-test-slave-kkrh"
server_key: "c06a94b566feaccc85ef4eac1030a5ed"
server_key_iv: "bcf042ba1b302cbd5c80759caabf1200"
server_key_version: "encryptionkey@0"
I20260928 02:43:14.211088 32293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:14.223080 32293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:14.224993 32293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:14.225169 32293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:14.229054 32293 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.62:35991
I20260928 02:43:14.229087 32360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.62:35991 every 8 connection(s)
I20260928 02:43:14.229430 32293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:14.230352 32361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:14.231809 32361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:43:14.234788 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 32293
I20260928 02:43:14.234902 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-0/wal/instance
I20260928 02:43:14.235122 9736 external_mini_cluster.cc:1428] Setting key ea40be9f4cd486e6afc564863a1a8fc7
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:43:14.241380 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.232640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.9.130.62:35991)
0928 02:43:14.232771 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.232775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.232801 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:43:14.233182 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.233184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.233200 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.233277 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.233937 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.234725 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.235810 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.236144 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.236263 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.238223 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.238229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.238229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.238243 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.239782 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.240231 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.240233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.240233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.240311 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.240538 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.240539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.240539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.240741 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.240834 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.241016 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.241158 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
I20260928 02:43:14.241384 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.232458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:35991 (local address 127.0.0.1:42696)
0928 02:43:14.232605 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.232638 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:14.233057 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:14.233334 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:14.233337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:14.233702 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:14.233854 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.233859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.234832 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.234833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.235715 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.235717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.236250 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.236251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.236314 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.236842 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:14.236850 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:14.238118 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:14.239936 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.239942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.239944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.240136 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.240390 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.240390 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.240391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.240473 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.240815 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:14.240820 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:14.240877 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 02:43:14.240984 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:43:14.241221 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":109,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for kudu/127.9.130.61@KRBTEST.COM; defaulting to no policy
I20260928 02:43:14.242084 32361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:43:14.242420 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.242291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:38775 (local address 127.0.0.1:51778)
0928 02:43:14.242343 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.242391 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 02:43:14.242558 32311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.130.61:38775: connect: Connection refused (error 111)
I20260928 02:43:14.242967 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.242887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:38775 (local address 127.0.0.1:51780)
0928 02:43:14.242928 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.242951 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:43:14.243072 32361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.61:38775: Network error: Client connection negotiation failed: client connection to 127.9.130.61:38775: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.61.keytab.
Entry for principal kudu/127.9.130.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.61.keytab.
I20260928 02:43:14.259181 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.61:38775
--webserver_interface=127.9.130.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.61.keytab
--principal=kudu/127.9.130.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:35991,127.9.130.61:38775,127.9.130.60:35755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:43:14.292294 32361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } attempt: 1
I20260928 02:43:14.292805 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.292672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:38775 (local address 127.0.0.1:51794)
0928 02:43:14.292739 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.292775 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:43:14.293183 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.293085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:38775 (local address 127.0.0.1:51810)
0928 02:43:14.293131 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.293156 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 02:43:14.293269 32361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.61:38775: Network error: Client connection negotiation failed: client connection to 127.9.130.61:38775: connect: Connection refused (error 111)
W20260928 02:43:14.337628 32368 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:43:14.337836 32368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:14.337855 32368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:14.337874 32368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:14.339035 32368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:14.339082 32368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:14.339097 32368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:14.339109 32368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:14.339121 32368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:14.340615 32368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:35991,127.9.130.61:38775,127.9.130.60:35755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.61.keytab
--principal=kudu/127.9.130.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.61:38775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.130.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:14.340870 32368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:14.341104 32368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:14.345994 32373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:14.346009 32374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:14.346077 32376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:14.346287 32368 server_base.cc:1034] running on GCE node
I20260928 02:43:14.346518 32368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:14.346809 32368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:14.347967 32368 hybrid_clock.cc:648] HybridClock initialized: now 1790563394347953 us; error 44 us; skew 500 ppm
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:14.350370 32368 init.cc:377] Logged in from keytab as kudu/127.9.130.61@KRBTEST.COM (short username kudu)
I20260928 02:43:14.350800 32368 webserver.cc:460] Webserver started at http://127.9.130.61:40355/ using document root <none> and password file <none>
I20260928 02:43:14.351018 32368 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:14.351084 32368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:14.351233 32368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:14.352324 32368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/data/instance:
uuid: "331c78003df944c9b22f2fc74a60b822"
format_stamp: "Formatted at 2026-09-28 02:43:14 on dist-test-slave-kkrh"
server_key: "4d2349964fee1f1977cfac88a52ec5c4"
server_key_iv: "a3da6687f7f7a5885bb03b6f76aa0207"
server_key_version: "encryptionkey@0"
I20260928 02:43:14.352684 32368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/wal/instance:
uuid: "331c78003df944c9b22f2fc74a60b822"
format_stamp: "Formatted at 2026-09-28 02:43:14 on dist-test-slave-kkrh"
server_key: "4d2349964fee1f1977cfac88a52ec5c4"
server_key_iv: "a3da6687f7f7a5885bb03b6f76aa0207"
server_key_version: "encryptionkey@0"
I20260928 02:43:14.354045 32368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:14.354771 32383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:14.354935 32368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:14.355031 32368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/wal
uuid: "331c78003df944c9b22f2fc74a60b822"
format_stamp: "Formatted at 2026-09-28 02:43:14 on dist-test-slave-kkrh"
server_key: "4d2349964fee1f1977cfac88a52ec5c4"
server_key_iv: "a3da6687f7f7a5885bb03b6f76aa0207"
server_key_version: "encryptionkey@0"
I20260928 02:43:14.355114 32368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:14.361491 32361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } attempt: 2
I20260928 02:43:14.361935 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.361800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:38775 (local address 127.0.0.1:51814)
0928 02:43:14.361860 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.361901 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:43:14.362293 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.362196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:38775 (local address 127.0.0.1:51824)
0928 02:43:14.362242 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.362271 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 02:43:14.362417 32361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.61:38775: Network error: Client connection negotiation failed: client connection to 127.9.130.61:38775: connect: Connection refused (error 111)
I20260928 02:43:14.365078 32368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:14.367264 32368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:14.367416 32368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:14.371347 32368 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.61:38775
I20260928 02:43:14.371378 32435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.61:38775 every 8 connection(s)
I20260928 02:43:14.371714 32368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/data/info.pb
I20260928 02:43:14.372722 32436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:14.373359 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 32368
I20260928 02:43:14.373468 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-1/wal/instance
I20260928 02:43:14.373656 9736 external_mini_cluster.cc:1428] Setting key 670963bc65c435335de586a28f04efee
I20260928 02:43:14.374284 32436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.61@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.130.60@KRBTEST.COM; defaulting to no policy
I20260928 02:43:14.383220 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.375023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.9.130.62:35991)
0928 02:43:14.375095 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.375097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.375312 (+ 215us) server_negotiation.cc:407] Connection header received
0928 02:43:14.375752 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.375754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.375769 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.375861 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.376483 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.377042 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.378021 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.378240 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.378316 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.380645 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.380647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.380648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.380656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.382033 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.382388 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.382389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.382389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.382438 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.382649 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.382649 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.382650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.382752 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.382801 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.383093 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.383156 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:43:14.383441 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.374999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:35991 (local address 127.0.0.1:42704)
0928 02:43:14.375178 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.375218 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:14.375661 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:14.375955 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:14.375958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:14.376201 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:14.376384 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.376388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.377136 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.377137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.377918 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.377920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.378334 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.378335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.378461 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.378966 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:14.378974 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:14.380546 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:14.382109 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.382115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.382116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.382323 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.382494 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.382494 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.382495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.382588 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.382816 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:14.382825 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:14.382980 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 02:43:14.383066 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:43:14.383184 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.130.60@KRBTEST.COM" created.
I20260928 02:43:14.384071 32436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.61@KRBTEST.COM for kudu/127.9.130.61@KRBTEST.COM
I20260928 02:43:14.391259 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.384344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:38775 (local address 127.0.0.1:51840)
0928 02:43:14.384545 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.384557 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:14.384613 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:14.384739 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:14.384740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:14.384901 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:14.384958 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.384959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.385764 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.385764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.386257 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.386258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.386753 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.386754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.386814 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.387237 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:14.387241 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:14.388424 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:14.390211 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.390211 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.390212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.390377 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.390604 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.390605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.390605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.390658 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.390955 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:14.390955 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:14.390992 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:43:14.391057 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:43:14.391151 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260928 02:43:14.391326 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.384414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51840 (local address 127.9.130.61:38775)
0928 02:43:14.384545 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.384549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.384610 (+ 61us) server_negotiation.cc:407] Connection header received
0928 02:43:14.384667 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.384668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.384678 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.384736 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.385023 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.385687 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.386354 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.386664 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.386753 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.388521 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.388529 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.388529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.388535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.390072 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.390477 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.390480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.390480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.390537 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.390727 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.390728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.390729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.390865 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.390975 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.391085 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.391203 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260928 02:43:14.391901 32436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:43:14.392279 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.392129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:35755 (local address 127.0.0.1:39696)
0928 02:43:14.392188 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.392254 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 02:43:14.392423 32387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
I20260928 02:43:14.392776 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.392681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:35755 (local address 127.0.0.1:39702)
0928 02:43:14.392732 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.392756 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:43:14.392922 32436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:35755: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.130.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.60.keytab.
Entry for principal kudu/127.9.130.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.60.keytab.
I20260928 02:43:14.398476 9736 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskL8zBe0/build/release/bin/kudu
/tmp/dist-test-taskL8zBe0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.60:35755
--webserver_interface=127.9.130.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.60.keytab
--principal=kudu/127.9.130.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.130.20:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.130.62:35991,127.9.130.61:38775,127.9.130.60:35755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:43:14.442181 32436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } attempt: 1
I20260928 02:43:14.442718 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.442563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:35755 (local address 127.0.0.1:39714)
0928 02:43:14.442633 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.442681 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 02:43:14.443135 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.443019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:35755 (local address 127.0.0.1:39724)
0928 02:43:14.443086 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.443114 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:43:14.443233 32436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:35755: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
I20260928 02:43:14.453577 32361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } attempt: 3
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for kudu/127.9.130.61@KRBTEST.COM
I20260928 02:43:14.460608 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.454003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.9.130.61:38775)
0928 02:43:14.454092 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.454094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.454107 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:43:14.454179 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.454180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.454191 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.454261 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.454586 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.455083 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.455843 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.456078 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.456155 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.458042 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.458044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.458044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.458053 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.459458 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.459812 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.459838 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.459839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.459897 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.460097 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.460098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.460099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.460210 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.460283 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.460441 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.460509 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:43:14.460608 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.453946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:38775 (local address 127.0.0.1:51842)
0928 02:43:14.454028 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.454041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:14.454110 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:14.454266 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:14.454267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:14.454430 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:14.454494 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.454496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.455168 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.455170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.455729 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.455731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.456166 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.456167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.456244 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.456714 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:14.456718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:14.457953 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:14.459568 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.459569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.459569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.459739 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.459964 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.459965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.459966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.460033 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.460275 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:14.460276 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:14.460323 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 02:43:14.460405 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:43:14.460507 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 02:43:14.461344 32361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:43:14.461685 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.461585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:35755 (local address 127.0.0.1:39738)
0928 02:43:14.461631 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.461660 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 02:43:14.462013 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.461922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:35755 (local address 127.0.0.1:39750)
0928 02:43:14.461970 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.461993 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 02:43:14.462108 32361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:35755: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
W20260928 02:43:14.479184 32443 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:43:14.479401 32443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:14.479429 32443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:14.479471 32443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:14.480686 32443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:14.480748 32443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:14.480772 32443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:14.480800 32443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:14.480830 32443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:14.482273 32443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.130.20:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.130.62:35991,127.9.130.61:38775,127.9.130.60:35755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/krb5kdc/kudu_127.9.130.60.keytab
--principal=kudu/127.9.130.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.130.60:35755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.130.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:14.482548 32443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:14.482769 32443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:14.487655 32448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:14.487759 32449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:14.487655 32451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:14.488095 32443 server_base.cc:1034] running on GCE node
I20260928 02:43:14.488365 32443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:14.488708 32443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:14.489882 32443 hybrid_clock.cc:648] HybridClock initialized: now 1790563394489857 us; error 40 us; skew 500 ppm
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:14.492954 32443 init.cc:377] Logged in from keytab as kudu/127.9.130.60@KRBTEST.COM (short username kudu)
I20260928 02:43:14.493219 32361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } attempt: 1
I20260928 02:43:14.493471 32443 webserver.cc:460] Webserver started at http://127.9.130.60:45195/ using document root <none> and password file <none>
I20260928 02:43:14.493606 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.493489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:35755 (local address 127.0.0.1:39766)
0928 02:43:14.493550 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.493575 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:43:14.493708 32443 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:14.493764 32443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:14.493892 32443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:14.493947 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.493864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:35755 (local address 127.0.0.1:39780)
0928 02:43:14.493899 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.493926 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 02:43:14.494031 32361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:35755: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
I20260928 02:43:14.495229 32443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/data/instance:
uuid: "b7131b8f04174f1aaabc77d486a21eb9"
format_stamp: "Formatted at 2026-09-28 02:43:14 on dist-test-slave-kkrh"
server_key: "58620ea86918bd47bbd4bf64e3352c4e"
server_key_iv: "03a5825eddfa40e2835f1f0f4a525cf6"
server_key_version: "encryptionkey@0"
I20260928 02:43:14.495639 32443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/wal/instance:
uuid: "b7131b8f04174f1aaabc77d486a21eb9"
format_stamp: "Formatted at 2026-09-28 02:43:14 on dist-test-slave-kkrh"
server_key: "58620ea86918bd47bbd4bf64e3352c4e"
server_key_iv: "03a5825eddfa40e2835f1f0f4a525cf6"
server_key_version: "encryptionkey@0"
I20260928 02:43:14.496980 32443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:14.497620 32458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:14.497804 32443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:14.497897 32443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/data,/tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/wal
uuid: "b7131b8f04174f1aaabc77d486a21eb9"
format_stamp: "Formatted at 2026-09-28 02:43:14 on dist-test-slave-kkrh"
server_key: "58620ea86918bd47bbd4bf64e3352c4e"
server_key_iv: "03a5825eddfa40e2835f1f0f4a525cf6"
server_key_version: "encryptionkey@0"
I20260928 02:43:14.497980 32443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:14.511364 32436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } attempt: 2
I20260928 02:43:14.511739 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.511620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:35755 (local address 127.0.0.1:39782)
0928 02:43:14.511677 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.511712 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:43:14.512099 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.511992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:35755 (local address 127.0.0.1:39786)
0928 02:43:14.512036 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.512082 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:43:14.512179 32436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.130.60:35755: Network error: Client connection negotiation failed: client connection to 127.9.130.60:35755: connect: Connection refused (error 111)
I20260928 02:43:14.514139 32443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:14.516376 32443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:14.516541 32443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:14.520301 32443 rpc_server.cc:307] RPC server started. Bound to: 127.9.130.60:35755
I20260928 02:43:14.520337 32510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.130.60:35755 every 8 connection(s)
I20260928 02:43:14.520677 32443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/data/info.pb
I20260928 02:43:14.521701 32511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:14.522863 9736 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskL8zBe0/build/release/bin/kudu as pid 32443
I20260928 02:43:14.522975 9736 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskL8zBe0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563127283700-9736-0/minicluster-data/master-2/wal/instance
I20260928 02:43:14.523166 9736 external_mini_cluster.cc:1428] Setting key 724824824332976d91fe954ec91f0664
I20260928 02:43:14.523563 32511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.60@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.60@KRBTEST.COM
I20260928 02:43:14.532591 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.524348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.9.130.62:35991)
0928 02:43:14.524400 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.524402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.524545 (+ 143us) server_negotiation.cc:407] Connection header received
0928 02:43:14.524692 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.524693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.524706 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.524772 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.525352 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.525879 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.526842 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.527064 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.527160 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.529493 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.529494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.529494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.529500 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.531051 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.531613 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.531613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.531614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.531662 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.531941 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.531942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.531942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.532043 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.532093 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.532405 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.532479 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:43:14.532732 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.524274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.62:35991 (local address 127.0.0.1:42710)
0928 02:43:14.524401 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.524451 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:14.524593 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:14.524793 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:14.524796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:14.525087 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:14.525245 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.525250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.525976 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.525977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.526744 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.526745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.527189 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.527191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.527272 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.527799 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:14.527807 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:14.529402 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:14.531150 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.531158 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.531160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.531533 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.531732 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.531732 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.531733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.531872 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.532109 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:14.532114 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:14.532262 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 02:43:14.532343 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:43:14.532489 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260928 02:43:14.533273 32511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:43:14.533661 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.523753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39796 (local address 127.9.130.60:35755)
0928 02:43:14.523940 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.523950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.523980 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:43:14.524559 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.524565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.524596 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.524716 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.525314 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.526085 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.526828 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.527139 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.527252 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.530037 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.530044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.530045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.530052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.532117 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.532444 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.532447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.532447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.532503 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.532731 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.532732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.532732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.533106 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.533253 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.533341 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.533412 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.60@KRBTEST.COM for kudu/127.9.130.61@KRBTEST.COM
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.62@KRBTEST.COM
I20260928 02:43:14.540093 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.533586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51846 (local address 127.9.130.61:38775)
0928 02:43:14.533647 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.533649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.533658 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:43:14.533683 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.533684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.533694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.533753 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.534070 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.534549 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.535349 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.535651 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.535737 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.537453 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.537454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.537455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.537462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.538936 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.539276 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.539277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.539278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.539329 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.539532 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.539533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.539534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.539643 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.539713 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.539902 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.539979 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:43:14.540340 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.533479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.61:38775 (local address 127.0.0.1:51846)
0928 02:43:14.533589 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.533598 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:14.533626 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:14.533800 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:14.533800 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:14.533941 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:14.533976 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.533977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.534633 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.534634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.535271 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.535273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.535835 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.535836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.535907 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.536284 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:14.536289 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:14.537387 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:14.539030 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.539031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.539031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.539194 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.539403 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.539404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.539404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.539459 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.539719 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:14.539720 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:14.539757 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:43:14.540169 (+ 412us) client_negotiation.cc:240] Negotiation successful
0928 02:43:14.540256 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 02:43:14.540727 32511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:43:14.541993 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.534887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42724 (local address 127.9.130.62:35991)
0928 02:43:14.534923 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.534925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.534932 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:43:14.534952 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.534953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.534961 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.535022 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.535398 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.536035 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.536658 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.536856 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.536941 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.539109 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.539110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.539110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.539117 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.540722 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.541240 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.541241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.541242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.541295 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.541500 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.541501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.541501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.541614 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.541780 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.541817 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.541883 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.60@KRBTEST.COM for kudu/127.9.130.60@KRBTEST.COM
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.130.61@KRBTEST.COM
I20260928 02:43:14.547358 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.540951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:35755 (local address 127.0.0.1:39808)
0928 02:43:14.541003 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.541020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:14.541057 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:14.541175 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:14.541176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:14.541300 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:14.541320 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.541321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.541898 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.541898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.542632 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.542634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.543111 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.543112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.543147 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.543460 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:14.543464 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:14.544747 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:14.546312 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.546313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.546314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.546513 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.546700 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.546701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.546701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.546760 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.547044 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:14.547045 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:14.547087 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:43:14.547170 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:43:14.547269 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 02:43:14.547477 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.541042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.9.130.60:35755)
0928 02:43:14.541085 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.541087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.541095 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:43:14.541111 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.541112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.541119 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.541172 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.541391 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.541823 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.542705 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.543037 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.543111 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.544825 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.544826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.544826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.544834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.546251 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.546592 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.546592 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.546593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.546639 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.546844 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.546845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.546846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.546969 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.547021 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.547186 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.547356 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":28}
I20260928 02:43:14.548684 32511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9: Bootstrap starting.
I20260928 02:43:14.549691 32511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:14.549958 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.542501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.9.130.61:38775)
0928 02:43:14.542539 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.542540 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.542548 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:43:14.542597 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.542598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.542606 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.542671 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.543227 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.543959 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.544869 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.545058 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.545138 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.547299 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.547300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.547300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.547306 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.548862 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.549249 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.549249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.549249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.549295 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.549449 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.549449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.549450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.549528 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.549643 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.549756 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.549844 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:43:14.550082 32511 log.cc:824] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:14.550985 32511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9: No bootstrap required, opened a new log
I20260928 02:43:14.551458 9736 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:14.552927 32511 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } }
I20260928 02:43:14.553117 32511 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7131b8f04174f1aaabc77d486a21eb9, State: Initialized, Role: FOLLOWER
I20260928 02:43:14.553217 32511 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } }
I20260928 02:43:14.553515 32511 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:14.553498 32517 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } } }
I20260928 02:43:14.552596 32335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7ef91b599c2f45efb3279c13bf4d3441"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42724
I20260928 02:43:14.553714 32517 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:43:14.555172 32528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:14.555440 32528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:14.555502 32443 master_runner.cc:186] Error getting master registration for 127.9.130.62:35991: OK, instance_id { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" instance_seqno: 1790563394227974 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:43:14.557096 32443 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:43:14.569161 32361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } attempt: 2
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.62@KRBTEST.COM for kudu/127.9.130.60@KRBTEST.COM
I20260928 02:43:14.576213 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.569466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:35755 (local address 127.0.0.1:39812)
0928 02:43:14.569611 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.569623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:14.569674 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:14.569819 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:14.569820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:14.569982 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:14.570024 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.570025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.570653 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.570654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.571159 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.571160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.571515 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.571516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.571572 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.572104 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:14.572109 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:14.573316 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:14.575085 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.575087 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.575087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.575297 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.575499 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.575499 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.575500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.575576 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.575905 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:14.575906 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:14.575950 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:43:14.576020 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:43:14.576112 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260928 02:43:14.576220 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.569565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.9.130.60:35755)
0928 02:43:14.569623 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.569625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.569657 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:43:14.569734 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.569735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.569745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.569816 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.570102 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.570568 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.571241 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.571424 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.571508 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.573407 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.573409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.573409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.573416 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.574990 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.575368 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.575369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.575370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.575430 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.575643 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.575644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.575644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.575810 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.575903 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.576054 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.576131 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:43:14.577737 32361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441: Bootstrap starting.
I20260928 02:43:14.578486 32361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:14.578774 32361 log.cc:824] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:14.579542 32361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441: No bootstrap required, opened a new log
I20260928 02:43:14.580664 32361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } }
I20260928 02:43:14.580806 32361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef91b599c2f45efb3279c13bf4d3441, State: Initialized, Role: FOLLOWER
I20260928 02:43:14.580893 32361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } }
I20260928 02:43:14.581213 32361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:14.581188 32531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } } }
I20260928 02:43:14.581271 32335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:43:14.581305 32531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:14.581353 32335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:14.582129 32335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } }
I20260928 02:43:14.582372 32335 leader_election.cc:288] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [CANDIDATE]: Term 1 election: Requested vote from peers 331c78003df944c9b22f2fc74a60b822 (127.9.130.61:38775), b7131b8f04174f1aaabc77d486a21eb9 (127.9.130.60:35755)
I20260928 02:43:14.582818 32485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ef91b599c2f45efb3279c13bf4d3441" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b7131b8f04174f1aaabc77d486a21eb9"
I20260928 02:43:14.582937 32485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9 [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:43:14.584142 32542 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:14.584183 32542 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:14.584218 32485 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef91b599c2f45efb3279c13bf4d3441 in term 1.
I20260928 02:43:14.584438 32312 leader_election.cc:302] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef91b599c2f45efb3279c13bf4d3441, b7131b8f04174f1aaabc77d486a21eb9; no voters:
I20260928 02:43:14.584558 32531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:14.584673 32531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [term 1 LEADER]: Becoming Leader. State: Replica: 7ef91b599c2f45efb3279c13bf4d3441, State: Running, Role: LEADER
I20260928 02:43:14.584744 32531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } }
I20260928 02:43:14.585099 32531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ef91b599c2f45efb3279c13bf4d3441. Latest consensus state: current_term: 1 leader_uuid: "7ef91b599c2f45efb3279c13bf4d3441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } } }
I20260928 02:43:14.585158 32531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:14.585350 32553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:14.585501 32553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:14.583971 32293 master_runner.cc:186] Error getting master registration for 127.9.130.61:38775: OK, instance_id { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" instance_seqno: 1790563394370251 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:43:14.586323 32293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:43:14.587730 32485 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9 [term 1 FOLLOWER]: Refusing update from remote peer 7ef91b599c2f45efb3279c13bf4d3441: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:14.588056 32531 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:14.589368 32517 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ef91b599c2f45efb3279c13bf4d3441. Latest consensus state: current_term: 1 leader_uuid: "7ef91b599c2f45efb3279c13bf4d3441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } } }
I20260928 02:43:14.589457 32517 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:14.597419 32531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ef91b599c2f45efb3279c13bf4d3441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } } }
I20260928 02:43:14.597522 32531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:14.597983 32517 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ef91b599c2f45efb3279c13bf4d3441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } } }
I20260928 02:43:14.598071 32517 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:14.598330 32554 mvcc.cc:205] Tried to move back new op lower bound from 7334147664228741120 to 7334147664219811840. Current Snapshot: MvccSnapshot[applied={T|T < 7334147664228741120}]
I20260928 02:43:14.599790 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.584489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.9.130.62:35991)
0928 02:43:14.584527 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.584529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.586837 (+ 2308us) server_negotiation.cc:407] Connection header received
0928 02:43:14.587079 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.587080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.587089 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.587117 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.590374 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.591102 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.593123 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.593414 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.593458 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.595371 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.595372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.595372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.595379 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.597363 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.598778 (+ 1415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.598779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.598779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.598844 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.599098 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.599099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.599100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.599246 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.599347 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.599593 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.599673 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:43:14.600170 32555 mvcc.cc:205] Tried to move back new op lower bound from 7334147664228741120 to 7334147664219811840. Current Snapshot: MvccSnapshot[applied={T|T < 7334147664228741120}]
I20260928 02:43:14.600733 32553 catalog_manager.cc:1347] Generated new cluster ID: d01f2e4634be4040903dae073c83dc4b
I20260928 02:43:14.600782 32553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:14.601228 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.586438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.9.130.61:38775)
0928 02:43:14.586478 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.586480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.591938 (+ 5458us) server_negotiation.cc:407] Connection header received
0928 02:43:14.592009 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.592010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.592023 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.592106 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.592765 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.593482 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.594693 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.595012 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.595085 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.597727 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.597728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.597728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.597735 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.599733 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.600350 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.600351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.600352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.600413 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.600629 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.600630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.600631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.600764 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.600847 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.601057 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.601129 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:43:14.601267 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.584700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.9.130.60:35755)
0928 02:43:14.584732 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.584734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.586307 (+ 1573us) server_negotiation.cc:407] Connection header received
0928 02:43:14.586985 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.586986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.586995 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.587022 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.590391 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.591110 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.593805 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.594099 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.594142 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.596425 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.596426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.596427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.596433 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.598696 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.599569 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.599570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.599570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.599626 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.600377 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.600378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.600379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.600536 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.600620 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.601100 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.601176 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:43:14.603302 32436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } attempt: 3
Sep 28 02:43:14 dist-test-slave-kkrh krb5kdc[32280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.130.61@KRBTEST.COM for kudu/127.9.130.60@KRBTEST.COM
I20260928 02:43:14.614012 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.603550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.130.60:35755 (local address 127.0.0.1:39834)
0928 02:43:14.603589 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:14.603600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:14.603641 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:14.603911 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:14.603912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:14.604065 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:14.604104 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.604105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.605839 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.605839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.606634 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:14.606636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.607160 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.607161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:14.607233 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.607630 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:14.607635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:14.609152 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:14.612612 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.612613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.612614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.612843 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.613199 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:14.613200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:14.613200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:14.613268 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:14.613706 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:14.613707 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:14.613762 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 02:43:14.613809 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:43:14.613898 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 02:43:14.615468 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:14.603722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.9.130.60:35755)
0928 02:43:14.603755 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.603757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.603763 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:43:14.603779 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.603780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.603788 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.603885 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.604211 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.604964 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.606743 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:14.607046 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:14.607100 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:14.609260 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:14.609262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:14.609262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:14.609268 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:14.612441 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.612984 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.612985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.612985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.613071 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:14.613342 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:14.613343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:14.613344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:14.613557 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:14.615098 (+ 1541us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:14.615168 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:43:14.615320 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:43:14.617219 32436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822: Bootstrap starting.
I20260928 02:43:14.617913 32436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:14.618181 32436 log.cc:824] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:14.624439 32436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822: No bootstrap required, opened a new log
I20260928 02:43:14.626495 32436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } }
I20260928 02:43:14.626667 32436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331c78003df944c9b22f2fc74a60b822, State: Initialized, Role: FOLLOWER
I20260928 02:43:14.626739 32436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } }
I20260928 02:43:14.627038 32436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:14.627959 32410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ef91b599c2f45efb3279c13bf4d3441" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "331c78003df944c9b22f2fc74a60b822"
I20260928 02:43:14.628062 32410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:14.628866 32410 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef91b599c2f45efb3279c13bf4d3441 in term 1.
I20260928 02:43:14.629258 32409 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822 [term 1 FOLLOWER]: Refusing update from remote peer 7ef91b599c2f45efb3279c13bf4d3441: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:14.629539 32548 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [LEADER]: Connected to new peer: Peer: permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:14.631569 32560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ef91b599c2f45efb3279c13bf4d3441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } } }
I20260928 02:43:14.631726 32560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:14.631880 32560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ef91b599c2f45efb3279c13bf4d3441. Latest consensus state: current_term: 1 leader_uuid: "7ef91b599c2f45efb3279c13bf4d3441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } } }
I20260928 02:43:14.632073 32560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:14.632164 32560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ef91b599c2f45efb3279c13bf4d3441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } } }
I20260928 02:43:14.632253 32560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:14.636891 32567 mvcc.cc:205] Tried to move back new op lower bound from 7334147664228741120 to 7334147664219811840. Current Snapshot: MvccSnapshot[applied={T|T < 7334147664228741120}]
W20260928 02:43:14.638056 32575 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:14.638149 32575 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:14.638504 32531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ef91b599c2f45efb3279c13bf4d3441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef91b599c2f45efb3279c13bf4d3441" member_type: VOTER last_known_addr { host: "127.9.130.62" port: 35991 } } peers { permanent_uuid: "331c78003df944c9b22f2fc74a60b822" member_type: VOTER last_known_addr { host: "127.9.130.61" port: 38775 } } peers { permanent_uuid: "b7131b8f04174f1aaabc77d486a21eb9" member_type: VOTER last_known_addr { host: "127.9.130.60" port: 35755 } } }
I20260928 02:43:14.638612 32531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:14.640887 32553 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:14.641574 32553 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:14.650996 32553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ef91b599c2f45efb3279c13bf4d3441: Generated new TSK 0
I20260928 02:43:14.651166 32553 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:43:14.678194 32514 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:14.676660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.9.130.60:35755)
0928 02:43:14.676701 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.676703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.677541 (+ 838us) server_negotiation.cc:407] Connection header received
0928 02:43:14.677641 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.677643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.677667 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.677697 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.678128 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39838: BlockingRecv error: recv got EOF from 127.0.0.1:39838 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 02:43:14.679320 32364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:14.677001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.9.130.62:35991)
0928 02:43:14.677040 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.677043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.678578 (+ 1535us) server_negotiation.cc:407] Connection header received
0928 02:43:14.678825 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.678826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.678839 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.678869 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.679260 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42742: BlockingRecv error: recv got EOF from 127.0.0.1:42742 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 02:43:14.680001 32440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:14.677280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.9.130.61:38775)
0928 02:43:14.677317 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.677319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.678761 (+ 1442us) server_negotiation.cc:407] Connection header received
0928 02:43:14.679590 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.679591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.679605 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.679640 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.679945 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51878: BlockingRecv error: recv got EOF from 127.0.0.1:51878 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 02:43:14.682591 32440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:14.681726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.9.130.61:38775)
0928 02:43:14.681758 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.681760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.681990 (+ 230us) server_negotiation.cc:407] Connection header received
0928 02:43:14.682094 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.682095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.682134 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.682164 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.682550 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51886: BlockingRecv error: recv got EOF from 127.0.0.1:51886 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 02:43:14.685549 32440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:14.684818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.9.130.61:38775)
0928 02:43:14.684851 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.684853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.685087 (+ 234us) server_negotiation.cc:407] Connection header received
0928 02:43:14.685179 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.685180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.685190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.685217 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.685495 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51898: BlockingRecv error: recv got EOF from 127.0.0.1:51898 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 02:43:14.689328 32440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:14.688597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.9.130.61:38775)
0928 02:43:14.688632 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.688633 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.688863 (+ 230us) server_negotiation.cc:407] Connection header received
0928 02:43:14.688958 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.688959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.688968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.688996 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.689290 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51902: BlockingRecv error: recv got EOF from 127.0.0.1:51902 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 02:43:14.695163 32440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:14.694411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.9.130.61:38775)
0928 02:43:14.694444 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.694446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.694694 (+ 248us) server_negotiation.cc:407] Connection header received
0928 02:43:14.694789 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.694790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.694799 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.694830 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.695124 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51904: BlockingRecv error: recv got EOF from 127.0.0.1:51904 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 02:43:14.705133 32440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:14.704319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.9.130.61:38775)
0928 02:43:14.704354 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.704356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.704623 (+ 267us) server_negotiation.cc:407] Connection header received
0928 02:43:14.704722 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.704723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.704733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.704761 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.705086 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51916: BlockingRecv error: recv got EOF from 127.0.0.1:51916 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 02:43:14.723417 32440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:14.722447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.9.130.61:38775)
0928 02:43:14.722490 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.722493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.722774 (+ 281us) server_negotiation.cc:407] Connection header received
0928 02:43:14.722874 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.722874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.722886 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.722921 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.723356 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51922: BlockingRecv error: recv got EOF from 127.0.0.1:51922 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 02:43:14.757488 32440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:14.756812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.9.130.61:38775)
0928 02:43:14.756896 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.756898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.756973 (+ 75us) server_negotiation.cc:407] Connection header received
0928 02:43:14.757041 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.757042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.757055 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.757117 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.757425 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51932: BlockingRecv error: recv got EOF from 127.0.0.1:51932 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 02:43:14.823282 32440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:14.822605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.9.130.61:38775)
0928 02:43:14.822656 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.822658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.822742 (+ 84us) server_negotiation.cc:407] Connection header received
0928 02:43:14.822808 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.822809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.822819 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.822862 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.823190 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51940: BlockingRecv error: recv got EOF from 127.0.0.1:51940 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 02:43:14.953243 32440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:14.952499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.9.130.61:38775)
0928 02:43:14.952618 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:14.952621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:14.952681 (+ 60us) server_negotiation.cc:407] Connection header received
0928 02:43:14.952730 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:14.952731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:14.952741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:14.952854 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:14.953187 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51954: BlockingRecv error: recv got EOF from 127.0.0.1:51954 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 02:43:15.211160 32440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:15.210469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.9.130.61:38775)
0928 02:43:15.210544 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:15.210546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:15.210559 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:43:15.210645 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:15.210646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:15.210659 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:15.210729 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:15.211068 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51968: BlockingRecv error: recv got EOF from 127.0.0.1:51968 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:43:15.555946 32528 catalog_manager.cc:1259] Loaded cluster ID: d01f2e4634be4040903dae073c83dc4b
I20260928 02:43:15.556017 32528 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9: loading cluster ID for follower catalog manager: success
I20260928 02:43:15.556824 32528 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9: acquiring CA information for follower catalog manager: success
I20260928 02:43:15.557122 32528 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7131b8f04174f1aaabc77d486a21eb9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:43:15.638602 32575 catalog_manager.cc:1259] Loaded cluster ID: d01f2e4634be4040903dae073c83dc4b
I20260928 02:43:15.638693 32575 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822: loading cluster ID for follower catalog manager: success
I20260928 02:43:15.639505 32575 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822: acquiring CA information for follower catalog manager: success
I20260928 02:43:15.639809 32575 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 331c78003df944c9b22f2fc74a60b822: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:43:15.727061 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:15.724431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.9.130.61:38775)
0928 02:43:15.724588 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:15.724590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:15.724603 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:43:15.724632 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:15.724633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:15.724680 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:15.724771 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:15.724931 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:15.725452 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:15.726200 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:15.726426 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:15.726492 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:15.726651 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:15.726757 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:15.726872 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:15.726938 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 02:43:15.726959 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260928 02:43:15.731389 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:15.728986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.9.130.60:35755)
0928 02:43:15.729087 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:15.729089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:15.729098 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:43:15.729176 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:15.729177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:15.729230 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:15.729314 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:15.729403 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:15.729884 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:15.730609 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:15.730827 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:15.730910 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:15.731059 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:15.731171 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:15.731259 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:15.731282 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 02:43:15.731302 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
I20260928 02:43:15.735673 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:15.733077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.9.130.60:35755)
0928 02:43:15.733119 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:15.733122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:15.733216 (+ 94us) server_negotiation.cc:407] Connection header received
0928 02:43:15.733282 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:15.733282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:15.733293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:15.733346 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:15.733454 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:15.734051 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:15.734991 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:15.735185 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:15.735266 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:15.735421 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:15.735451 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:15.735510 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:15.735569 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:43:15.735589 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:43:15.736440 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:15.733181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.9.130.61:38775)
0928 02:43:15.733220 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:15.733222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:15.733579 (+ 357us) server_negotiation.cc:407] Connection header received
0928 02:43:15.733608 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:15.733609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:15.733618 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:15.733676 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:15.733781 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:15.734507 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:15.735650 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:15.735967 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:15.736018 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:15.736208 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:15.736241 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:15.736294 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:15.736334 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 02:43:15.736355 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:43:15.736725 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:15.733027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.9.130.62:35991)
0928 02:43:15.733143 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:15.733145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:15.733183 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:43:15.733302 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:15.733303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:15.733356 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:15.733455 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:15.733714 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:15.734479 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:15.735644 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:15.735970 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:15.736023 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:15.736247 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:15.736362 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:15.736432 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:15.736616 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 02:43:15.736644 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260928 02:43:15.738409 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 32293
I20260928 02:43:15.743640 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 32368
I20260928 02:43:15.748371 9736 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskL8zBe0/build/release/bin/kudu with pid 32443
2026-09-28T02:43:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2789 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5594 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (268472 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST